2022-03-08T22:55:06.870 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-08T22:55:06.872 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} --archive /home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727300 -- /home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727300/orig.config.yaml 2022-03-08T22:55:06.906 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-08T22:55:06.975 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727300 branch: wip-yuri5-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} email: null first_in_suite: false job_id: '6727300' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2022-03-07-0958 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri5-testing-2022-03-07-0958 sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri5-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 targets: smithi107.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/r8NZE8YhcMPJDMJ2+YDVSnDodzqI7o1lVGmWYfZ7EQu8PjMXwm/MmbUReJIz4rCAR9/0R7TVNHDS8wMEX6oR0Nw1aU32ScRpWKLkETkmJG8tVJ0mTIInhxeALcpVozdRzLxA07vA0A6WVg2d/Cx1Bd/ENeA+abYYI+KSgbHqOGUkVytF3YxZrv9IQsMiaPIoRS+DluwN+6AdshGwW3Y9+bT+SdXgd3cb7iC640JwHjPQXvlR6+ehx96cRjRqchKANdzqT/d3zJe9Kd8MHbut1o7qBPvfw9yCwUFVshDgipKQUJ+F6FFNgyL49MHkg6JOP+Ws9UQ1rVR44zNnSpIniN74BhNo577kQVrQKLf39ZIOScca8VA+G2GjzyaxOX+tjh4nevVoQeilsS9ed/a7z3Z08eJ8GSypOwe0k2uKWOrw0pa5Wqpfzw5JaCxFZ9PFvJsv29BBDACZG33MLZ05PD6edFLRmDzHbEHrH0kJEr7I/O7rPYnD0rOcQOFzJwk= smithi155.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2QPLqIIYvrSlDjIMM2rLRnRHD/G4UIswyCzOzOV/tYRrRXwEAqfEH3zpKyoTffMTORMZhw6D+nFllOEUWVqrYbG6b8zMstepNGXoIbMnK5aVUAwj46ByE6XPbT3EgGMgwFomKpk4uSU3j3MNW8/bGQOp4AXK/rz2BcUARkHZxDd+ix2NtAExLo3/VlcDjlRucMp3GCztJRx5s0Q+WxJjTXBZsM+w7TsBh7uXWr3sp82qJznJvdc4RFBds2VGzkl4soH6HGndA/LWT8kH1g01kYZ4yaeHkjhBOx8WJb1eR2ZZaKu5eayo4ACfIRcK5L9c5zh8lfraeyMkSzxkkxS3jrD+gcEJ7WD/qpP3yYb63TQCedrYVtk62B/VfZXscjIBmigVieKULvoELwwz9bb+KUrrUsUafHRpHl48nPZVLRSnu9BfW+xbmNbMPpG82wHOauSRZN2o/nDHOlUjFCfzuarCOWgGus0jmXiizij8nTel5ZEKN8FTtaFAfgBDA+FU= smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC9SfZxGp8x7oFpKJju4R+gc2oMbRY4hW9HOkRYe+FfDvI6sEZxZR/e2NXn7eRE4e3ZLMNa7hEvtG2J2CdFbfjG3DTora8rzPdY85Vo4r55QiNreJEJgcHBqq+WSSqNz5RNy2abIDskEwfnel/OQuK2RerRpF0Is0BFcuOCIm2biZXuDxt7iI1dKz0HHg/g99otIyozUKfFUOn/dKXJp6mhxxgd7McGkVmBuLyXvRObOZ/aWOICQ9o1JhRukUPGTtujxXlGsGphzbjpwWj9ruQifWhWciPJM7I6eO1SvtU7P7CMy5+kd9SSVTmL4sJAJELVHHI/GeOwMxmMAqQ4vfMdW7p0JusiKoi8Y44LkNdko9hTNDLXHXCFKH6S2V/2WJcsL6S3W2f/HrgMaPn3Xskb4qp27FxhKZ6DqozF1NI95FUDnuWF1+SiyULenO8IeVhSliX/yT+Uoig/8rCLGErgYNKke1oPMCbsbEt2sgergYj3BWpnlhvAq7F01U2tj0= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-08_22:38:39 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-08T22:55:06.975 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa; will attempt to use it 2022-03-08T22:55:06.976 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks 2022-03-08T22:55:06.976 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-08T22:55:06.978 INFO:teuthology.task.internal:Checking packages... 2022-03-08T22:55:07.005 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7a44e9ebca556862e9704be295a07719d7db47b3' 2022-03-08T22:55:07.005 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-08T22:55:07.006 INFO:teuthology.packaging:ref: None 2022-03-08T22:55:07.006 INFO:teuthology.packaging:tag: None 2022-03-08T22:55:07.007 INFO:teuthology.packaging:branch: wip-yuri5-testing-2022-03-07-0958 2022-03-08T22:55:07.007 INFO:teuthology.packaging:sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T22:55:07.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2022-03-07-0958 2022-03-08T22:55:07.167 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11025-g7a44e9eb-1focal 2022-03-08T22:55:07.168 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-08T22:55:07.176 INFO:teuthology.task.internal:no buildpackages task found 2022-03-08T22:55:07.176 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-08T22:55:07.210 INFO:teuthology.task.internal:Saving configuration 2022-03-08T22:55:07.224 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-08T22:55:07.242 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-08T22:55:07.268 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727300', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:49:32.382665', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHLms4JpoOry2SrhHY+k8lPCF8tgPzwDH9MujorAoLeo'} 2022-03-08T22:55:07.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727300', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:49:32.381511', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOycXrahnT72myXNrRmQeIqwTV53Fn7ib9c6KZhZ8hYm'} 2022-03-08T22:55:07.311 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-08_22:38:39-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6727300', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 22:49:32.380029', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa+LOaeOOr3nxqnmLQLad4lztyJF/kOjDZqrelHNdNU'} 2022-03-08T22:55:07.311 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-08T22:55:07.319 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-08T22:55:07.320 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-08T22:55:07.320 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-08T22:55:07.320 INFO:teuthology.run_tasks:Running task console_log... 2022-03-08T22:55:07.637 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1a219b29d8>, signals=[15]) 2022-03-08T22:55:07.639 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-08T22:55:07.649 INFO:teuthology.task.internal:Opening connections... 2022-03-08T22:55:07.649 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2022-03-08T22:55:07.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:55:07.706 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2022-03-08T22:55:07.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:55:07.760 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2022-03-08T22:55:07.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:55:07.810 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-08T22:55:07.820 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2022-03-08T22:55:07.828 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2022-03-08T22:55:07.828 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2022-03-08T22:55:07.878 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="Ubuntu" 2022-03-08T22:55:07.878 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:55:07.878 INFO:teuthology.orchestra.run.smithi107.stdout:ID=ubuntu 2022-03-08T22:55:07.879 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE=debian 2022-03-08T22:55:07.879 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:55:07.879 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="20.04" 2022-03-08T22:55:07.879 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:55:07.880 INFO:teuthology.orchestra.run.smithi107.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:55:07.880 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:55:07.880 INFO:teuthology.orchestra.run.smithi107.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:55:07.880 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_CODENAME=focal 2022-03-08T22:55:07.880 INFO:teuthology.orchestra.run.smithi107.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:55:07.881 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2022-03-08T22:55:07.900 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2022-03-08T22:55:07.907 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2022-03-08T22:55:07.908 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2022-03-08T22:55:07.958 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Ubuntu" 2022-03-08T22:55:07.958 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:55:07.959 INFO:teuthology.orchestra.run.smithi155.stdout:ID=ubuntu 2022-03-08T22:55:07.959 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE=debian 2022-03-08T22:55:07.959 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:55:07.959 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="20.04" 2022-03-08T22:55:07.960 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:55:07.960 INFO:teuthology.orchestra.run.smithi155.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:55:07.960 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:55:07.960 INFO:teuthology.orchestra.run.smithi155.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:55:07.960 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_CODENAME=focal 2022-03-08T22:55:07.961 INFO:teuthology.orchestra.run.smithi155.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:55:07.961 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2022-03-08T22:55:07.978 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2022-03-08T22:55:07.986 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2022-03-08T22:55:07.986 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2022-03-08T22:55:08.034 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Ubuntu" 2022-03-08T22:55:08.035 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T22:55:08.035 INFO:teuthology.orchestra.run.smithi187.stdout:ID=ubuntu 2022-03-08T22:55:08.035 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE=debian 2022-03-08T22:55:08.035 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T22:55:08.036 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="20.04" 2022-03-08T22:55:08.036 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T22:55:08.036 INFO:teuthology.orchestra.run.smithi187.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T22:55:08.036 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T22:55:08.036 INFO:teuthology.orchestra.run.smithi187.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T22:55:08.037 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_CODENAME=focal 2022-03-08T22:55:08.037 INFO:teuthology.orchestra.run.smithi187.stdout:UBUNTU_CODENAME=focal 2022-03-08T22:55:08.037 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2022-03-08T22:55:08.056 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-08T22:55:08.067 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-08T22:55:08.106 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-08T22:55:08.106 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:55:08.109 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:55:08.111 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2022-03-08T22:55:08.117 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-08T22:55:08.137 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-08T22:55:08.138 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:55:08.158 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:55:08.160 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2022-03-08T22:55:08.186 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-08T22:55:08.269 INFO:teuthology.run_tasks:Running task kernel... 2022-03-08T22:55:08.285 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-08T22:55:08.286 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-08T22:55:08.286 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-08T22:55:08.286 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-08T22:55:08.287 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2022-03-08T22:55:08.288 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2022-03-08T22:55:08.288 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2022-03-08T22:55:08.296 INFO:teuthology.orchestra.run.smithi107.stdout:5.4.0-97-generic 2022-03-08T22:55:08.296 INFO:teuthology.orchestra.run.smithi155.stdout:5.4.0-97-generic 2022-03-08T22:55:08.296 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.4.0-97-generic 2022-03-08T22:55:08.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get clean 2022-03-08T22:55:08.297 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.4.0-97-generic 2022-03-08T22:55:08.298 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get clean 2022-03-08T22:55:08.299 INFO:teuthology.orchestra.run.smithi187.stdout:5.4.0-97-generic 2022-03-08T22:55:08.299 INFO:teuthology.task.kernel:Running kernel on smithi187: 5.4.0-97-generic 2022-03-08T22:55:08.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-get clean 2022-03-08T22:55:08.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-get update 2022-03-08T22:55:08.372 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2022-03-08T22:55:08.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2022-03-08T22:55:08.524 INFO:teuthology.orchestra.run.smithi187.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:55:08.634 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:55:08.658 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:55:08.661 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:55:08.716 INFO:teuthology.orchestra.run.smithi187.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:55:08.719 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:55:08.800 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:55:08.800 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T22:55:08.808 INFO:teuthology.orchestra.run.smithi187.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:55:08.850 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T22:55:08.943 INFO:teuthology.orchestra.run.smithi187.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:55:08.944 INFO:teuthology.orchestra.run.smithi187.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:55:08.947 INFO:teuthology.orchestra.run.smithi187.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:55:08.947 INFO:teuthology.orchestra.run.smithi187.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:55:08.955 INFO:teuthology.orchestra.run.smithi187.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:55:08.956 INFO:teuthology.orchestra.run.smithi187.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:55:08.957 INFO:teuthology.orchestra.run.smithi187.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:55:08.958 INFO:teuthology.orchestra.run.smithi187.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:55:08.963 INFO:teuthology.orchestra.run.smithi187.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:55:08.969 INFO:teuthology.orchestra.run.smithi187.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:55:08.974 INFO:teuthology.orchestra.run.smithi187.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:55:08.986 INFO:teuthology.orchestra.run.smithi187.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:55:08.987 INFO:teuthology.orchestra.run.smithi187.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:55:08.987 INFO:teuthology.orchestra.run.smithi187.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:55:08.987 INFO:teuthology.orchestra.run.smithi187.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:55:09.053 INFO:teuthology.orchestra.run.smithi187.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:55:09.064 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:55:09.148 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:55:09.177 INFO:teuthology.orchestra.run.smithi187.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:55:09.186 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T22:55:09.261 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:55:09.289 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:55:09.403 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:55:09.432 INFO:teuthology.orchestra.run.smithi187.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:55:09.536 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-08T22:55:09.553 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-08T22:55:09.555 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-08T22:55:09.569 INFO:teuthology.orchestra.run.smithi155.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:55:09.574 INFO:teuthology.orchestra.run.smithi187.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:55:09.584 INFO:teuthology.orchestra.run.smithi155.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:55:09.585 INFO:teuthology.orchestra.run.smithi155.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:55:09.593 INFO:teuthology.orchestra.run.smithi187.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:55:09.594 INFO:teuthology.orchestra.run.smithi187.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:55:09.595 INFO:teuthology.orchestra.run.smithi187.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:55:09.622 INFO:teuthology.orchestra.run.smithi187.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:55:09.627 INFO:teuthology.orchestra.run.smithi187.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:55:09.627 INFO:teuthology.orchestra.run.smithi187.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:55:09.636 INFO:teuthology.orchestra.run.smithi155.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:55:09.637 INFO:teuthology.orchestra.run.smithi155.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:55:09.641 INFO:teuthology.orchestra.run.smithi155.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:55:09.641 INFO:teuthology.orchestra.run.smithi155.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:55:09.659 INFO:teuthology.orchestra.run.smithi187.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:55:09.671 INFO:teuthology.orchestra.run.smithi155.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:55:09.681 INFO:teuthology.orchestra.run.smithi187.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:55:09.686 INFO:teuthology.orchestra.run.smithi187.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:55:09.693 INFO:teuthology.orchestra.run.smithi155.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:55:09.695 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-08T22:55:09.696 INFO:teuthology.orchestra.run.smithi155.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:55:09.699 INFO:teuthology.orchestra.run.smithi155.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:55:09.709 INFO:teuthology.orchestra.run.smithi187.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:55:09.710 INFO:teuthology.orchestra.run.smithi187.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:55:09.711 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T22:55:09.711 INFO:teuthology.orchestra.run.smithi187.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:55:09.711 INFO:teuthology.orchestra.run.smithi187.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:55:09.712 INFO:teuthology.orchestra.run.smithi187.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:55:09.712 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T22:55:09.713 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-08T22:55:09.714 INFO:teuthology.orchestra.run.smithi155.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:55:09.715 INFO:teuthology.orchestra.run.smithi155.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:55:09.718 INFO:teuthology.orchestra.run.smithi155.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:55:09.738 INFO:teuthology.orchestra.run.smithi155.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:55:09.739 INFO:teuthology.orchestra.run.smithi155.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:55:09.740 INFO:teuthology.orchestra.run.smithi155.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:55:09.740 INFO:teuthology.orchestra.run.smithi155.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:55:09.742 INFO:teuthology.orchestra.run.smithi187.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:55:09.749 INFO:teuthology.orchestra.run.smithi155.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:55:09.755 INFO:teuthology.orchestra.run.smithi155.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:55:09.755 INFO:teuthology.orchestra.run.smithi155.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:55:09.766 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-08T22:55:09.774 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T22:55:09.775 INFO:teuthology.orchestra.run.smithi107.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T22:55:09.803 INFO:teuthology.orchestra.run.smithi155.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:55:09.803 INFO:teuthology.orchestra.run.smithi107.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-08T22:55:09.810 INFO:teuthology.orchestra.run.smithi107.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T22:55:09.830 INFO:teuthology.orchestra.run.smithi107.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-08T22:55:09.840 INFO:teuthology.orchestra.run.smithi107.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T22:55:09.841 INFO:teuthology.orchestra.run.smithi107.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-08T22:55:09.842 INFO:teuthology.orchestra.run.smithi107.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T22:55:09.842 INFO:teuthology.orchestra.run.smithi155.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:55:09.852 INFO:teuthology.orchestra.run.smithi187.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:55:09.852 INFO:teuthology.orchestra.run.smithi187.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:55:09.853 INFO:teuthology.orchestra.run.smithi187.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:55:09.853 INFO:teuthology.orchestra.run.smithi187.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:55:09.854 INFO:teuthology.orchestra.run.smithi187.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:55:09.949 INFO:teuthology.orchestra.run.smithi107.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T22:55:09.950 INFO:teuthology.orchestra.run.smithi107.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-08T22:55:09.951 INFO:teuthology.orchestra.run.smithi107.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T22:55:09.952 INFO:teuthology.orchestra.run.smithi107.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [909 kB] 2022-03-08T22:55:09.962 INFO:teuthology.orchestra.run.smithi107.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-08T22:55:09.973 INFO:teuthology.orchestra.run.smithi107.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-08T22:55:09.975 INFO:teuthology.orchestra.run.smithi107.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:55:09.976 INFO:teuthology.orchestra.run.smithi155.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T22:55:09.976 INFO:teuthology.orchestra.run.smithi155.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:55:09.977 INFO:teuthology.orchestra.run.smithi107.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T22:55:09.977 INFO:teuthology.orchestra.run.smithi107.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T22:55:09.977 INFO:teuthology.orchestra.run.smithi107.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T22:55:09.977 INFO:teuthology.orchestra.run.smithi107.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T22:55:09.978 INFO:teuthology.orchestra.run.smithi107.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T22:55:09.979 INFO:teuthology.orchestra.run.smithi155.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:55:09.979 INFO:teuthology.orchestra.run.smithi155.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:55:09.979 INFO:teuthology.orchestra.run.smithi155.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:55:09.979 INFO:teuthology.orchestra.run.smithi155.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:55:09.980 INFO:teuthology.orchestra.run.smithi155.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:55:09.980 INFO:teuthology.orchestra.run.smithi155.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:55:09.980 INFO:teuthology.orchestra.run.smithi155.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:55:10.004 INFO:teuthology.orchestra.run.smithi107.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T22:55:10.004 INFO:teuthology.orchestra.run.smithi107.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T22:55:10.056 INFO:teuthology.orchestra.run.smithi107.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T22:55:10.057 INFO:teuthology.orchestra.run.smithi107.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T22:55:10.057 INFO:teuthology.orchestra.run.smithi107.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T22:55:10.058 INFO:teuthology.orchestra.run.smithi155.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:55:10.058 INFO:teuthology.orchestra.run.smithi107.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T22:55:10.058 INFO:teuthology.orchestra.run.smithi107.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T22:55:10.087 INFO:teuthology.orchestra.run.smithi107.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T22:55:10.087 INFO:teuthology.orchestra.run.smithi107.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T22:55:10.088 INFO:teuthology.orchestra.run.smithi107.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:55:10.139 INFO:teuthology.orchestra.run.smithi155.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T22:55:10.140 INFO:teuthology.orchestra.run.smithi155.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:55:10.167 INFO:teuthology.orchestra.run.smithi107.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T22:55:12.356 INFO:teuthology.orchestra.run.smithi187.stdout:Fetched 10.2 MB in 2s (6,036 kB/s) 2022-03-08T22:55:12.785 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 10.2 MB in 2s (4,700 kB/s) 2022-03-08T22:55:12.814 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 10.2 MB in 2s (4,428 kB/s) 2022-03-08T22:55:13.506 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2022-03-08T22:55:13.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:55:13.600 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2022-03-08T22:55:13.756 INFO:teuthology.orchestra.run.smithi187.stdout:Building dependency tree... 2022-03-08T22:55:13.756 INFO:teuthology.orchestra.run.smithi187.stdout:Reading state information... 2022-03-08T22:55:13.889 INFO:teuthology.orchestra.run.smithi187.stdout:The following additional packages will be installed: 2022-03-08T22:55:13.889 INFO:teuthology.orchestra.run.smithi187.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:55:13.889 INFO:teuthology.orchestra.run.smithi187.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:55:13.890 INFO:teuthology.orchestra.run.smithi187.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:55:13.891 INFO:teuthology.orchestra.run.smithi187.stdout:Suggested packages: 2022-03-08T22:55:13.891 INFO:teuthology.orchestra.run.smithi187.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:55:13.891 INFO:teuthology.orchestra.run.smithi187.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:55:13.933 INFO:teuthology.orchestra.run.smithi187.stdout:The following NEW packages will be installed: 2022-03-08T22:55:13.933 INFO:teuthology.orchestra.run.smithi187.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:55:13.934 INFO:teuthology.orchestra.run.smithi187.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:55:13.934 INFO:teuthology.orchestra.run.smithi187.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:55:13.934 INFO:teuthology.orchestra.run.smithi187.stdout: thermald upower usbmuxd 2022-03-08T22:55:13.977 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-03-08T22:55:14.001 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:55:14.030 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2022-03-08T22:55:14.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:55:14.065 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-03-08T22:55:14.121 INFO:teuthology.orchestra.run.smithi187.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:55:14.121 INFO:teuthology.orchestra.run.smithi187.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:55:14.121 INFO:teuthology.orchestra.run.smithi187.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:55:14.122 INFO:teuthology.orchestra.run.smithi187.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:55:14.122 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2022-03-08T22:55:14.220 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-03-08T22:55:14.221 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-03-08T22:55:14.294 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2022-03-08T22:55:14.295 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2022-03-08T22:55:14.351 INFO:teuthology.orchestra.run.smithi155.stdout:The following additional packages will be installed: 2022-03-08T22:55:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:55:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:55:14.351 INFO:teuthology.orchestra.run.smithi155.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:55:14.352 INFO:teuthology.orchestra.run.smithi155.stdout:Suggested packages: 2022-03-08T22:55:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:55:14.352 INFO:teuthology.orchestra.run.smithi155.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:55:14.405 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2022-03-08T22:55:14.406 INFO:teuthology.orchestra.run.smithi155.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:55:14.407 INFO:teuthology.orchestra.run.smithi155.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:55:14.407 INFO:teuthology.orchestra.run.smithi155.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:55:14.407 INFO:teuthology.orchestra.run.smithi155.stdout: thermald upower usbmuxd 2022-03-08T22:55:14.427 INFO:teuthology.orchestra.run.smithi187.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:55:14.427 INFO:teuthology.orchestra.run.smithi187.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:55:14.435 INFO:teuthology.orchestra.run.smithi187.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:55:14.443 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2022-03-08T22:55:14.443 INFO:teuthology.orchestra.run.smithi107.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:55:14.444 INFO:teuthology.orchestra.run.smithi107.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-08T22:55:14.444 INFO:teuthology.orchestra.run.smithi107.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-08T22:55:14.445 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2022-03-08T22:55:14.445 INFO:teuthology.orchestra.run.smithi107.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T22:55:14.445 INFO:teuthology.orchestra.run.smithi107.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-08T22:55:14.492 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2022-03-08T22:55:14.493 INFO:teuthology.orchestra.run.smithi107.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T22:55:14.493 INFO:teuthology.orchestra.run.smithi107.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-08T22:55:14.494 INFO:teuthology.orchestra.run.smithi107.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-08T22:55:14.494 INFO:teuthology.orchestra.run.smithi107.stdout: thermald upower usbmuxd 2022-03-08T22:55:14.531 INFO:teuthology.orchestra.run.smithi187.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:55:14.544 INFO:teuthology.orchestra.run.smithi187.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:55:14.589 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:55:14.589 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:55:14.589 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:55:14.590 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:55:14.681 INFO:teuthology.orchestra.run.smithi107.stdout:0 upgraded, 13 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:55:14.681 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 80.1 MB of archives. 2022-03-08T22:55:14.682 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T22:55:14.682 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T22:55:14.876 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:55:14.913 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:55:14.913 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:55:15.024 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T22:55:15.121 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:55:15.122 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T22:55:15.123 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:55:15.123 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T22:55:15.146 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T22:55:15.217 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-08T22:55:15.558 INFO:teuthology.orchestra.run.smithi187.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:55:15.837 INFO:teuthology.orchestra.run.smithi187.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:55:16.029 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:55:16.166 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-08T22:55:16.361 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:55:16.524 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-08T22:55:18.066 INFO:teuthology.orchestra.run.smithi187.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:55:18.066 INFO:teuthology.orchestra.run.smithi187.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:55:18.067 INFO:teuthology.orchestra.run.smithi187.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:55:18.072 INFO:teuthology.orchestra.run.smithi187.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:55:18.075 INFO:teuthology.orchestra.run.smithi187.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:55:18.549 INFO:teuthology.orchestra.run.smithi187.stdout:Fetched 80.1 MB in 4s (19.1 MB/s) 2022-03-08T22:55:18.732 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:55:18.759 INFO:teuthology.orchestra.run.smithi155.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:55:18.760 INFO:teuthology.orchestra.run.smithi155.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:55:18.761 INFO:teuthology.orchestra.run.smithi155.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:55:18.766 INFO:teuthology.orchestra.run.smithi155.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:55:18.769 INFO:teuthology.orchestra.run.smithi155.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:55:18.784 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-08T22:55:18.784 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T22:55:18.785 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T22:55:18.791 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T22:55:18.793 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T22:55:19.184 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 80.1 MB in 4s (18.4 MB/s) 2022-03-08T22:55:19.190 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 80.1 MB in 4s (18.6 MB/s) 2022-03-08T22:55:19.375 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:55:19.381 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T22:55:20.201 INFO:teuthology.orchestra.run.smithi187.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:55:20.207 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:55:20.363 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:55:20.757 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:55:20.772 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:55:20.816 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:55:20.898 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:55:20.904 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:55:20.915 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T22:55:20.919 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T22:55:21.045 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:55:21.046 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:55:21.144 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:55:21.157 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:55:21.194 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:55:21.349 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:55:21.364 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:55:21.399 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:55:21.422 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T22:55:21.436 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T22:55:21.481 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:55:21.514 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:55:21.529 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:55:21.564 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:55:21.745 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:55:21.759 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:55:21.792 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T22:55:21.802 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:55:21.805 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T22:55:21.866 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:55:21.891 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:55:21.905 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:55:21.941 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:55:22.170 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:55:22.172 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T22:55:22.185 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:55:22.185 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T22:55:22.220 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:55:22.221 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:55:22.270 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:55:22.284 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:55:22.328 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:22.548 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:55:22.557 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T22:55:22.560 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:55:22.572 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T22:55:22.597 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:55:22.607 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:55:22.910 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:55:22.924 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:55:22.933 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-08T22:55:22.948 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:55:22.969 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:22.992 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:24.521 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:55:24.534 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:55:24.634 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:25.116 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:55:25.129 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:55:25.157 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:25.193 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:55:25.209 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:55:25.300 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:25.534 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-08T22:55:25.550 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:55:25.645 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:25.796 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:55:25.811 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:55:25.854 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:26.145 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-08T22:55:26.159 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-08T22:55:26.203 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:30.969 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:55:30.987 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:55:31.027 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:55:31.349 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:55:31.364 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:55:31.408 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:55:31.743 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:55:31.761 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:55:31.761 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package thermald. 2022-03-08T22:55:31.774 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:55:31.801 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:55:31.810 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:55:32.146 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:55:32.161 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:55:32.204 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:55:32.234 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T22:55:32.252 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-08T22:55:32.292 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:55:32.318 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package upower. 2022-03-08T22:55:32.331 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:55:32.387 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:55:32.565 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T22:55:32.580 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T22:55:32.584 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package thermald. 2022-03-08T22:55:32.597 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:55:32.626 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:55:32.640 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:55:32.809 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:55:32.826 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:55:32.868 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:55:32.976 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package thermald. 2022-03-08T22:55:32.991 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T22:55:33.034 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:55:33.126 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:55:33.159 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package upower. 2022-03-08T22:55:33.175 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:55:33.227 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:55:33.248 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:55:33.391 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:55:33.517 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:55:33.600 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package upower. 2022-03-08T22:55:33.613 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T22:55:33.643 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:55:33.668 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T22:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:55:33.769 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:55:33.870 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:55:33.998 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:55:34.033 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T22:55:34.045 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T22:55:34.099 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:55:34.138 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:55:34.256 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:55:34.323 INFO:teuthology.orchestra.run.smithi187.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:55:34.373 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T22:55:34.382 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:55:34.388 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:55:34.495 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T22:55:34.508 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:55:34.559 INFO:teuthology.orchestra.run.smithi187.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:55:34.568 INFO:teuthology.orchestra.run.smithi187.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:55:34.569 INFO:teuthology.orchestra.run.smithi187.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:55:34.626 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:55:34.637 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T22:55:34.764 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T22:55:34.777 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:55:34.890 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T22:55:35.016 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T22:55:35.088 INFO:teuthology.orchestra.run.smithi187.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:55:35.192 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T22:55:35.274 INFO:teuthology.orchestra.run.smithi107.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:55:35.306 INFO:teuthology.orchestra.run.smithi187.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:55:35.329 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:55:35.342 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:55:35.504 INFO:teuthology.orchestra.run.smithi107.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:55:35.510 INFO:teuthology.orchestra.run.smithi107.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:55:35.510 INFO:teuthology.orchestra.run.smithi107.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:55:35.650 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:55:35.650 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:55:35.699 INFO:teuthology.orchestra.run.smithi155.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T22:55:35.751 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T22:55:35.925 INFO:teuthology.orchestra.run.smithi155.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T22:55:35.934 INFO:teuthology.orchestra.run.smithi155.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T22:55:35.934 INFO:teuthology.orchestra.run.smithi155.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T22:55:36.004 INFO:teuthology.orchestra.run.smithi107.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:55:36.122 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:36.247 INFO:teuthology.orchestra.run.smithi107.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:55:36.316 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:55:36.410 INFO:teuthology.orchestra.run.smithi155.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T22:55:36.601 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:55:36.601 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:55:36.648 INFO:teuthology.orchestra.run.smithi155.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T22:55:36.689 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T22:55:36.993 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:55:36.993 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T22:55:37.121 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:37.383 INFO:teuthology.orchestra.run.smithi187.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:55:37.383 INFO:teuthology.orchestra.run.smithi187.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:55:37.485 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:37.549 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:38.407 INFO:teuthology.orchestra.run.smithi107.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:55:38.407 INFO:teuthology.orchestra.run.smithi107.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:55:38.581 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:38.748 INFO:teuthology.orchestra.run.smithi155.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-08T22:55:38.749 INFO:teuthology.orchestra.run.smithi155.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-08T22:55:38.850 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:38.903 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:39.899 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:40.229 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:40.268 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:55:40.394 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:55:41.226 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:55:41.352 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:55:41.548 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-08T22:55:41.674 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T22:55:44.358 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:55:45.157 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:55:45.478 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T22:55:45.652 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:55:45.752 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:45.953 INFO:teuthology.orchestra.run.smithi187.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:55:45.954 INFO:teuthology.orchestra.run.smithi187.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:55:46.240 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:55:46.442 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T22:55:46.543 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:46.739 INFO:teuthology.orchestra.run.smithi107.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:55:46.740 INFO:teuthology.orchestra.run.smithi107.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:55:46.848 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-08T22:55:47.037 INFO:teuthology.orchestra.run.smithi155.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T22:55:47.037 INFO:teuthology.orchestra.run.smithi155.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:56:09.371 INFO:teuthology.orchestra.run.smithi187.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:56:09.372 INFO:teuthology.orchestra.run.smithi187.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:56:09.384 INFO:teuthology.orchestra.run.smithi187.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:56:09.408 INFO:teuthology.orchestra.run.smithi187.stdout:Generating grub configuration file ... 2022-03-08T22:56:09.698 INFO:teuthology.orchestra.run.smithi187.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:56:09.726 INFO:teuthology.orchestra.run.smithi187.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:56:10.075 INFO:teuthology.orchestra.run.smithi187.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:56:10.087 INFO:teuthology.orchestra.run.smithi187.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:56:10.315 INFO:teuthology.orchestra.run.smithi107.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:56:10.316 INFO:teuthology.orchestra.run.smithi107.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:56:10.332 INFO:teuthology.orchestra.run.smithi107.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:56:10.360 INFO:teuthology.orchestra.run.smithi107.stdout:Generating grub configuration file ... 2022-03-08T22:56:10.622 INFO:teuthology.orchestra.run.smithi155.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T22:56:10.622 INFO:teuthology.orchestra.run.smithi155.stdout:Sourcing file `/etc/default/grub' 2022-03-08T22:56:10.639 INFO:teuthology.orchestra.run.smithi155.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:56:10.642 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:56:10.660 INFO:teuthology.orchestra.run.smithi155.stdout:Generating grub configuration file ... 2022-03-08T22:56:10.677 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:56:10.983 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:56:11.005 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:56:11.082 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:56:11.094 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:56:11.386 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:56:11.398 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:56:12.669 INFO:teuthology.orchestra.run.smithi187.stdout:done 2022-03-08T22:56:13.022 DEBUG:teuthology.orchestra.run.smithi187:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:56:13.056 INFO:teuthology.orchestra.run.smithi187.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:56:13.057 INFO:teuthology.orchestra.run.smithi187.stdout:Status: install ok installed 2022-03-08T22:56:13.057 INFO:teuthology.orchestra.run.smithi187.stdout:Priority: optional 2022-03-08T22:56:13.057 INFO:teuthology.orchestra.run.smithi187.stdout:Section: kernel 2022-03-08T22:56:13.058 INFO:teuthology.orchestra.run.smithi187.stdout:Installed-Size: 19 2022-03-08T22:56:13.058 INFO:teuthology.orchestra.run.smithi187.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:56:13.058 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: amd64 2022-03-08T22:56:13.058 INFO:teuthology.orchestra.run.smithi187.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:56:13.058 INFO:teuthology.orchestra.run.smithi187.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:56:13.059 INFO:teuthology.orchestra.run.smithi187.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:56:13.059 INFO:teuthology.orchestra.run.smithi187.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:56:13.059 INFO:teuthology.orchestra.run.smithi187.stdout:Recommends: thermald 2022-03-08T22:56:13.059 INFO:teuthology.orchestra.run.smithi187.stdout:Description: Generic Linux kernel image 2022-03-08T22:56:13.060 INFO:teuthology.orchestra.run.smithi187.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:56:13.060 INFO:teuthology.orchestra.run.smithi187.stdout: available. 2022-03-08T22:56:13.060 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:56:13.060 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:56:13.061 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:56:13.061 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-08T22:56:13.061 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:56:13.062 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-get clean 2022-03-08T22:56:13.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt-get update 2022-03-08T22:56:13.406 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:56:13.406 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:56:13.476 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:56:13.561 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:56:13.711 INFO:teuthology.orchestra.run.smithi107.stdout:done 2022-03-08T22:56:13.985 INFO:teuthology.orchestra.run.smithi155.stdout:done 2022-03-08T22:56:14.113 DEBUG:teuthology.orchestra.run.smithi107:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:56:14.145 INFO:teuthology.orchestra.run.smithi107.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:56:14.145 INFO:teuthology.orchestra.run.smithi107.stdout:Status: install ok installed 2022-03-08T22:56:14.145 INFO:teuthology.orchestra.run.smithi107.stdout:Priority: optional 2022-03-08T22:56:14.146 INFO:teuthology.orchestra.run.smithi107.stdout:Section: kernel 2022-03-08T22:56:14.146 INFO:teuthology.orchestra.run.smithi107.stdout:Installed-Size: 19 2022-03-08T22:56:14.146 INFO:teuthology.orchestra.run.smithi107.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:56:14.146 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: amd64 2022-03-08T22:56:14.147 INFO:teuthology.orchestra.run.smithi107.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:56:14.147 INFO:teuthology.orchestra.run.smithi107.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:56:14.147 INFO:teuthology.orchestra.run.smithi107.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:56:14.147 INFO:teuthology.orchestra.run.smithi107.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:56:14.148 INFO:teuthology.orchestra.run.smithi107.stdout:Recommends: thermald 2022-03-08T22:56:14.148 INFO:teuthology.orchestra.run.smithi107.stdout:Description: Generic Linux kernel image 2022-03-08T22:56:14.148 INFO:teuthology.orchestra.run.smithi107.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:56:14.148 INFO:teuthology.orchestra.run.smithi107.stdout: available. 2022-03-08T22:56:14.148 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:56:14.149 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:56:14.149 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:56:14.149 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-08T22:56:14.149 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:56:14.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get clean 2022-03-08T22:56:14.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2022-03-08T22:56:14.334 DEBUG:teuthology.orchestra.run.smithi155:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:56:14.341 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:56:14.362 INFO:teuthology.orchestra.run.smithi155.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:56:14.362 INFO:teuthology.orchestra.run.smithi155.stdout:Status: install ok installed 2022-03-08T22:56:14.363 INFO:teuthology.orchestra.run.smithi155.stdout:Priority: optional 2022-03-08T22:56:14.363 INFO:teuthology.orchestra.run.smithi155.stdout:Section: kernel 2022-03-08T22:56:14.363 INFO:teuthology.orchestra.run.smithi155.stdout:Installed-Size: 19 2022-03-08T22:56:14.363 INFO:teuthology.orchestra.run.smithi155.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:56:14.363 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: amd64 2022-03-08T22:56:14.364 INFO:teuthology.orchestra.run.smithi155.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:56:14.364 INFO:teuthology.orchestra.run.smithi155.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:56:14.364 INFO:teuthology.orchestra.run.smithi155.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:56:14.364 INFO:teuthology.orchestra.run.smithi155.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:56:14.364 INFO:teuthology.orchestra.run.smithi155.stdout:Recommends: thermald 2022-03-08T22:56:14.365 INFO:teuthology.orchestra.run.smithi155.stdout:Description: Generic Linux kernel image 2022-03-08T22:56:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:56:14.365 INFO:teuthology.orchestra.run.smithi155.stdout: available. 2022-03-08T22:56:14.365 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-08T22:56:14.366 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T22:56:14.366 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T22:56:14.366 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-08T22:56:14.366 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T22:56:14.367 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get clean 2022-03-08T22:56:14.438 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2022-03-08T22:56:14.453 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:56:14.535 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:56:14.638 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:56:14.720 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T22:56:14.720 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T22:56:14.809 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T22:56:14.914 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T22:56:14.931 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2022-03-08T22:56:14.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:56:15.021 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2022-03-08T22:56:15.176 INFO:teuthology.orchestra.run.smithi187.stdout:Building dependency tree... 2022-03-08T22:56:15.177 INFO:teuthology.orchestra.run.smithi187.stdout:Reading state information... 2022-03-08T22:56:15.345 INFO:teuthology.orchestra.run.smithi187.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:56:15.345 INFO:teuthology.orchestra.run.smithi187.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:56:15.347 DEBUG:teuthology.orchestra.run.smithi187:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:56:15.367 INFO:teuthology.orchestra.run.smithi187.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:56:15.367 INFO:teuthology.orchestra.run.smithi187.stdout:Status: install ok installed 2022-03-08T22:56:15.367 INFO:teuthology.orchestra.run.smithi187.stdout:Priority: optional 2022-03-08T22:56:15.368 INFO:teuthology.orchestra.run.smithi187.stdout:Section: kernel 2022-03-08T22:56:15.368 INFO:teuthology.orchestra.run.smithi187.stdout:Installed-Size: 19 2022-03-08T22:56:15.368 INFO:teuthology.orchestra.run.smithi187.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:56:15.368 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: amd64 2022-03-08T22:56:15.369 INFO:teuthology.orchestra.run.smithi187.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:56:15.369 INFO:teuthology.orchestra.run.smithi187.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:56:15.369 INFO:teuthology.orchestra.run.smithi187.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:56:15.369 INFO:teuthology.orchestra.run.smithi187.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:56:15.369 INFO:teuthology.orchestra.run.smithi187.stdout:Recommends: thermald 2022-03-08T22:56:15.370 INFO:teuthology.orchestra.run.smithi187.stdout:Description: Generic Linux kernel image 2022-03-08T22:56:15.370 INFO:teuthology.orchestra.run.smithi187.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:56:15.370 INFO:teuthology.orchestra.run.smithi187.stdout: available. 2022-03-08T22:56:15.370 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2022-03-08T22:56:15.375 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.KSzZ5cng4Z 2022-03-08T22:56:15.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub/grub.cfg /tmp/tmp.KSzZ5cng4Z 2022-03-08T22:56:15.434 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.KSzZ5cng4Z 2022-03-08T22:56:15.463 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.KSzZ5cng4Z is 8KB 2022-03-08T22:56:15.478 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.KSzZ5cng4Z 2022-03-08T22:56:15.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:56:15.541 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T22:56:15.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:56:15.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:56:15.574 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:56:15.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo update-grub 2022-03-08T22:56:15.891 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2022-03-08T22:56:15.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:56:15.982 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2022-03-08T22:56:16.137 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2022-03-08T22:56:16.138 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2022-03-08T22:56:16.264 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-03-08T22:56:16.285 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T22:56:16.307 INFO:teuthology.orchestra.run.smithi107.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:56:16.308 INFO:teuthology.orchestra.run.smithi107.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:56:16.310 DEBUG:teuthology.orchestra.run.smithi107:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:56:16.326 INFO:teuthology.orchestra.run.smithi107.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:56:16.326 INFO:teuthology.orchestra.run.smithi107.stdout:Status: install ok installed 2022-03-08T22:56:16.327 INFO:teuthology.orchestra.run.smithi107.stdout:Priority: optional 2022-03-08T22:56:16.327 INFO:teuthology.orchestra.run.smithi107.stdout:Section: kernel 2022-03-08T22:56:16.327 INFO:teuthology.orchestra.run.smithi107.stdout:Installed-Size: 19 2022-03-08T22:56:16.327 INFO:teuthology.orchestra.run.smithi107.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:56:16.327 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: amd64 2022-03-08T22:56:16.328 INFO:teuthology.orchestra.run.smithi107.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:56:16.328 INFO:teuthology.orchestra.run.smithi107.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:56:16.328 INFO:teuthology.orchestra.run.smithi107.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:56:16.328 INFO:teuthology.orchestra.run.smithi107.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:56:16.329 INFO:teuthology.orchestra.run.smithi107.stdout:Recommends: thermald 2022-03-08T22:56:16.329 INFO:teuthology.orchestra.run.smithi107.stdout:Description: Generic Linux kernel image 2022-03-08T22:56:16.329 INFO:teuthology.orchestra.run.smithi107.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:56:16.329 INFO:teuthology.orchestra.run.smithi107.stdout: available. 2022-03-08T22:56:16.330 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2022-03-08T22:56:16.360 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-03-08T22:56:16.376 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.cSzRQuhXJZ 2022-03-08T22:56:16.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub/grub.cfg /tmp/tmp.cSzRQuhXJZ 2022-03-08T22:56:16.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.cSzRQuhXJZ 2022-03-08T22:56:16.468 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.cSzRQuhXJZ is 8KB 2022-03-08T22:56:16.483 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.cSzRQuhXJZ 2022-03-08T22:56:16.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:56:16.533 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-03-08T22:56:16.534 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-03-08T22:56:16.545 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T22:56:16.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:56:16.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:56:16.608 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:56:16.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo update-grub 2022-03-08T22:56:16.705 INFO:teuthology.orchestra.run.smithi155.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-08T22:56:16.705 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded. 2022-03-08T22:56:16.707 DEBUG:teuthology.orchestra.run.smithi155:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T22:56:16.725 INFO:teuthology.orchestra.run.smithi155.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T22:56:16.726 INFO:teuthology.orchestra.run.smithi155.stdout:Status: install ok installed 2022-03-08T22:56:16.726 INFO:teuthology.orchestra.run.smithi155.stdout:Priority: optional 2022-03-08T22:56:16.726 INFO:teuthology.orchestra.run.smithi155.stdout:Section: kernel 2022-03-08T22:56:16.727 INFO:teuthology.orchestra.run.smithi155.stdout:Installed-Size: 19 2022-03-08T22:56:16.727 INFO:teuthology.orchestra.run.smithi155.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T22:56:16.727 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: amd64 2022-03-08T22:56:16.727 INFO:teuthology.orchestra.run.smithi155.stdout:Source: linux-meta-hwe-5.13 2022-03-08T22:56:16.727 INFO:teuthology.orchestra.run.smithi155.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-08T22:56:16.728 INFO:teuthology.orchestra.run.smithi155.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T22:56:16.728 INFO:teuthology.orchestra.run.smithi155.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T22:56:16.728 INFO:teuthology.orchestra.run.smithi155.stdout:Recommends: thermald 2022-03-08T22:56:16.728 INFO:teuthology.orchestra.run.smithi155.stdout:Description: Generic Linux kernel image 2022-03-08T22:56:16.728 INFO:teuthology.orchestra.run.smithi155.stdout: This package will always depend on the latest generic kernel image 2022-03-08T22:56:16.729 INFO:teuthology.orchestra.run.smithi155.stdout: available. 2022-03-08T22:56:16.729 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2022-03-08T22:56:16.752 INFO:teuthology.orchestra.run.smithi187.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:56:16.753 INFO:teuthology.orchestra.run.smithi187.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:56:16.772 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.P9m3i3cVBT 2022-03-08T22:56:16.772 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub/grub.cfg /tmp/tmp.P9m3i3cVBT 2022-03-08T22:56:16.787 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.P9m3i3cVBT 2022-03-08T22:56:16.807 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2022-03-08T22:56:16.829 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.P9m3i3cVBT is 8KB 2022-03-08T22:56:16.844 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.P9m3i3cVBT 2022-03-08T22:56:16.851 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T22:56:16.909 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T22:56:16.909 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T22:56:16.909 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T22:56:16.950 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-08T22:56:16.950 DEBUG:teuthology.orchestra.run.smithi155:> sudo update-grub 2022-03-08T22:56:17.082 INFO:teuthology.orchestra.run.smithi187.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:56:17.093 INFO:teuthology.orchestra.run.smithi187.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:56:17.492 INFO:teuthology.orchestra.run.smithi187.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:56:17.504 INFO:teuthology.orchestra.run.smithi187.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:56:17.778 INFO:teuthology.orchestra.run.smithi107.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:56:17.787 INFO:teuthology.orchestra.run.smithi107.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:56:17.809 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2022-03-08T22:56:18.056 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:56:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:56:18.071 INFO:teuthology.orchestra.run.smithi155.stderr:Sourcing file `/etc/default/grub' 2022-03-08T22:56:18.073 INFO:teuthology.orchestra.run.smithi155.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T22:56:18.092 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2022-03-08T22:56:18.368 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-08T22:56:18.379 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-08T22:56:18.461 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:56:18.472 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:56:18.780 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T22:56:18.791 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T22:56:19.121 INFO:teuthology.orchestra.run.smithi187.stderr:done 2022-03-08T22:56:19.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2022-03-08T22:56:20.073 INFO:teuthology.orchestra.run.smithi107.stderr:done 2022-03-08T22:56:20.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2022-03-08T22:56:20.418 INFO:teuthology.orchestra.run.smithi155.stderr:done 2022-03-08T22:56:20.420 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2022-03-08T22:56:49.154 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-08T22:56:49.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2022-03-08T22:56:49.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:56:50.078 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-08T22:56:50.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2022-03-08T22:56:50.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:56:50.423 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-08T22:56:50.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-03-08T22:56:50.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:57:07.531 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2022-03-08T22:57:08.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2022-03-08T22:57:08.815 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2022-03-08T22:57:37.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2022-03-08T22:57:37.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:57:38.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2022-03-08T22:57:38.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:57:38.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-03-08T22:57:38.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:57:53.867 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2022-03-08T22:57:56.939 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2022-03-08T22:57:57.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2022-03-08T22:58:23.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2022-03-08T22:58:23.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:58:24.024 DEBUG:teuthology.orchestra.run.smithi187:> true 2022-03-08T22:58:24.475 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2022-03-08T22:58:24.475 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-35-generic"... 2022-03-08T22:58:24.475 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2022-03-08T22:58:24.522 INFO:teuthology.orchestra.run.smithi187.stdout:5.13.0-35-generic 2022-03-08T22:58:24.523 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:58:24.523 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:58:24.523 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:58:25.525 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-08T22:58:25.525 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:58:25.542 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2022-03-08T22:58:25.587 DEBUG:teuthology.parallel:result is None 2022-03-08T22:58:26.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2022-03-08T22:58:26.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:58:27.135 DEBUG:teuthology.orchestra.run.smithi107:> true 2022-03-08T22:58:27.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-03-08T22:58:27.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T22:58:27.642 DEBUG:teuthology.orchestra.run.smithi155:> true 2022-03-08T22:58:27.904 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2022-03-08T22:58:27.904 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-08T22:58:27.904 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2022-03-08T22:58:27.953 INFO:teuthology.orchestra.run.smithi107.stdout:5.13.0-35-generic 2022-03-08T22:58:27.953 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:58:27.953 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:58:27.954 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:58:28.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-03-08T22:58:28.118 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-35-generic"... 2022-03-08T22:58:28.119 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2022-03-08T22:58:28.125 INFO:teuthology.orchestra.run.smithi155.stdout:5.13.0-35-generic 2022-03-08T22:58:28.126 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-08T22:58:28.126 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T22:58:28.126 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T22:58:28.955 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-08T22:58:28.955 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:58:29.022 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2022-03-08T22:58:29.049 DEBUG:teuthology.parallel:result is None 2022-03-08T22:58:29.127 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-08T22:58:29.128 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T22:58:29.192 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2022-03-08T22:58:29.224 DEBUG:teuthology.parallel:result is None 2022-03-08T22:58:29.224 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-08T22:58:29.234 INFO:teuthology.task.internal:Creating test directory... 2022-03-08T22:58:29.234 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:58:29.238 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:58:29.241 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T22:58:29.250 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-08T22:58:29.280 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-08T22:58:29.314 INFO:teuthology.task.internal:Creating archive directory... 2022-03-08T22:58:29.315 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:58:29.318 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:58:29.320 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T22:58:29.344 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-08T22:58:29.355 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-08T22:58:29.355 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:58:29.358 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:58:29.361 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T22:58:29.379 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:58:29.385 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:58:29.386 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:58:29.387 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:58:29.393 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:58:29.395 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T22:58:29.396 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-08T22:58:29.403 INFO:teuthology.task.internal:Configuring sudo... 2022-03-08T22:58:29.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:58:29.433 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:58:29.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T22:58:29.459 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-08T22:58:29.470 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-08T22:58:29.470 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:58:29.473 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:58:29.475 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T22:58:29.506 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:58:29.514 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:58:29.565 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T22:58:29.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:58:29.601 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:58:29.611 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:58:29.660 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T22:58:29.661 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:58:29.679 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T22:58:29.688 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T22:58:29.698 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T22:58:29.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T22:58:29.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2022-03-08T22:58:29.769 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2022-03-08T22:58:29.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2022-03-08T22:58:29.816 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-08T22:58:29.827 INFO:teuthology.task.internal:Starting timer... 2022-03-08T22:58:29.828 INFO:teuthology.run_tasks:Running task pcp... 2022-03-08T22:58:29.893 INFO:teuthology.run_tasks:Running task selinux... 2022-03-08T22:58:29.933 DEBUG:teuthology.task.selinux:Excluding smithi107: OS 'ubuntu' does not support SELinux 2022-03-08T22:58:29.933 DEBUG:teuthology.task.selinux:Excluding smithi155: OS 'ubuntu' does not support SELinux 2022-03-08T22:58:29.933 DEBUG:teuthology.task.selinux:Excluding smithi187: OS 'ubuntu' does not support SELinux 2022-03-08T22:58:29.934 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-08T22:58:29.934 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-08T22:58:29.934 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-08T22:58:29.935 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-08T22:58:29.966 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-08T22:58:29.967 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-08T22:58:29.986 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-08T22:58:29.988 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi155.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-08T23:01:37.674 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2022-03-08T23:01:37.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2022-03-08T23:01:37.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T23:01:37.723 DEBUG:teuthology.orchestra.run.smithi107:> true 2022-03-08T23:01:37.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-03-08T23:01:37.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T23:01:37.883 DEBUG:teuthology.orchestra.run.smithi155:> true 2022-03-08T23:01:38.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2022-03-08T23:01:38.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T23:01:38.057 DEBUG:teuthology.orchestra.run.smithi187:> true 2022-03-08T23:01:38.180 INFO:teuthology.run_tasks:Running task clock... 2022-03-08T23:01:38.191 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-08T23:01:38.192 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T23:01:38.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T23:01:38.197 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T23:01:38.197 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T23:01:38.200 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T23:01:38.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T23:01:38.229 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T23:01:38.230 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: Command line: ntpd -gq 2022-03-08T23:01:38.231 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: proto: precision = 0.087 usec (-23) 2022-03-08T23:01:38.231 INFO:teuthology.orchestra.run.smithi107.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T23:01:38.232 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T23:01:38.232 INFO:teuthology.orchestra.run.smithi107.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T23:01:38.233 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T23:01:38.234 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T23:01:38.234 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T23:01:38.234 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T23:01:38.234 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: Listen normally on 3 enp3s0f1 172.21.15.107:123 2022-03-08T23:01:38.235 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: Listen normally on 4 lo [::1]:123 2022-03-08T23:01:38.235 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:47eb%5]:123 2022-03-08T23:01:38.235 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:38 ntpd[9021]: Listening on routing socket on fd #22 for interface updates 2022-03-08T23:01:38.236 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T23:01:38.236 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: Command line: ntpd -gq 2022-03-08T23:01:38.236 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: proto: precision = 0.093 usec (-23) 2022-03-08T23:01:38.237 INFO:teuthology.orchestra.run.smithi155.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T23:01:38.237 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T23:01:38.238 INFO:teuthology.orchestra.run.smithi155.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T23:01:38.239 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T23:01:38.239 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T23:01:38.239 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T23:01:38.240 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T23:01:38.240 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: Listen normally on 3 enp3s0f1 172.21.15.155:123 2022-03-08T23:01:38.240 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: Listen normally on 4 lo [::1]:123 2022-03-08T23:01:38.240 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6ded%5]:123 2022-03-08T23:01:38.241 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:38 ntpd[9083]: Listening on routing socket on fd #22 for interface updates 2022-03-08T23:01:38.252 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T23:01:38.252 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: Command line: ntpd -gq 2022-03-08T23:01:38.253 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: proto: precision = 0.058 usec (-24) 2022-03-08T23:01:38.253 INFO:teuthology.orchestra.run.smithi187.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T23:01:38.254 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T23:01:38.254 INFO:teuthology.orchestra.run.smithi187.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T23:01:38.256 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T23:01:38.256 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T23:01:38.256 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T23:01:38.256 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T23:01:38.257 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: Listen normally on 3 enp3s0f1 172.21.15.187:123 2022-03-08T23:01:38.257 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: Listen normally on 4 lo [::1]:123 2022-03-08T23:01:38.257 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77b1%5]:123 2022-03-08T23:01:38.257 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:38 ntpd[9135]: Listening on routing socket on fd #22 for interface updates 2022-03-08T23:01:39.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:39.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:39.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:39.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:39.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:39.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:39.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:39.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:39.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:39.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:39.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:39.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:39.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:39.236 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:39.236 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:39.236 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:39.236 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:39.236 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:39 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:40.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:40 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:40.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:40 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:40.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:40 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:40.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:40 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:40.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:40 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:40.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:40 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:40.233 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:40.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:40.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:40.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:40.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:40.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:40.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:40.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:40.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:40.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:40.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:40.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:40 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:41.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:41.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:41.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:41.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:41.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:41.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:41 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:41.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:41 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:41 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:41 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:41 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:41 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:41.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:41 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:41.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:41 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:41 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:41 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:41.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:41 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:41.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:41 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:41.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:41 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:42.230 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:42 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:42 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:42 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:42 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:42 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:42.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:42 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:42.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:42.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:42.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:42.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:42.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:42.238 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:42 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:42.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:42 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:42 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:42 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:42.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:42 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:42.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:42 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:42.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:42 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:43.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:43.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:43.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:43.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:43.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.235 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:43.236 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:43 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:43.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:43 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:43 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:43 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:43 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.238 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:43 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:43.238 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:43 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:43.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:43.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:43.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:43.258 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:43.258 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:43.258 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:43.258 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:43.259 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:43 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:44.230 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:44 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:44 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:44 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:44 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:44 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:44.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:44 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:44.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:44.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:44.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:44.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:44.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.237 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:44.238 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:44 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:44.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:44 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:44 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:44 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:44.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:44 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:44.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:44 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:44.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:44 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:45.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.231 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.232 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:45.233 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:45.234 INFO:teuthology.orchestra.run.smithi107.stdout: 8 Mar 23:01:45 ntpd[9021]: ntpd: time slew -0.010245 s 2022-03-08T23:01:45.234 INFO:teuthology.orchestra.run.smithi107.stdout:ntpd: time slew -0.010245s 2022-03-08T23:01:45.234 INFO:teuthology.orchestra.run.smithi107.stderr: 8 Mar 23:01:45 ntpd[9021]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T23:01:45.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:45 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:45 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:45 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:45 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:45 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:45.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:45 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:45.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:45.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:45.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:45.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:45 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:45.293 INFO:teuthology.orchestra.run.smithi107.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T23:01:45.294 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================== 2022-03-08T23:01:45.294 INFO:teuthology.orchestra.run.smithi107.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:45.294 INFO:teuthology.orchestra.run.smithi107.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:45.294 INFO:teuthology.orchestra.run.smithi107.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:45.295 INFO:teuthology.orchestra.run.smithi107.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:46.233 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:46.233 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:46.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:46.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:46.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:46.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:46.234 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:46.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:46.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:46.235 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:46.236 INFO:teuthology.orchestra.run.smithi155.stdout: 8 Mar 23:01:46 ntpd[9083]: ntpd: time slew -0.010464 s 2022-03-08T23:01:46.236 INFO:teuthology.orchestra.run.smithi155.stdout:ntpd: time slew -0.010464s 2022-03-08T23:01:46.236 INFO:teuthology.orchestra.run.smithi155.stderr: 8 Mar 23:01:46 ntpd[9083]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T23:01:46.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:46 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:46.253 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:46 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:46.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:46 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:46.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:46 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:46.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:46 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:46.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:46 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T23:01:46.302 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================== 2022-03-08T23:01:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:47.254 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:47.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:47.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:47.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:47.255 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:47.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:47.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:47.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:47.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:47.256 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:47.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:47.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:47.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:47.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:47.257 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T23:01:47.258 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T23:01:47.258 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T23:01:47.258 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T23:01:47.259 INFO:teuthology.orchestra.run.smithi187.stdout: 8 Mar 23:01:47 ntpd[9135]: ntpd: time slew +0.009351 s 2022-03-08T23:01:47.259 INFO:teuthology.orchestra.run.smithi187.stdout:ntpd: time slew +0.009351s 2022-03-08T23:01:47.259 INFO:teuthology.orchestra.run.smithi187.stderr: 8 Mar 23:01:47 ntpd[9135]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T23:01:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T23:01:47.312 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================== 2022-03-08T23:01:47.313 INFO:teuthology.orchestra.run.smithi187.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:47.313 INFO:teuthology.orchestra.run.smithi187.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:47.313 INFO:teuthology.orchestra.run.smithi187.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:47.313 INFO:teuthology.orchestra.run.smithi187.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T23:01:47.314 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-08T23:01:47.330 INFO:tasks.kubeadm:Kubeadm start 2022-03-08T23:01:47.330 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-08T23:01:47.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T23:01:47.334 DEBUG:teuthology.orchestra.run.smithi155:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T23:01:47.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T23:01:47.355 INFO:teuthology.orchestra.run.smithi107.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T23:01:47.358 INFO:teuthology.orchestra.run.smithi155.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T23:01:47.362 INFO:teuthology.orchestra.run.smithi107.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T23:01:47.363 INFO:teuthology.orchestra.run.smithi187.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T23:01:47.364 INFO:teuthology.orchestra.run.smithi155.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T23:01:47.370 INFO:teuthology.orchestra.run.smithi155.stdout:net.ipv4.ip_forward = 1 2022-03-08T23:01:47.370 INFO:teuthology.orchestra.run.smithi107.stdout:net.ipv4.ip_forward = 1 2022-03-08T23:01:47.372 INFO:teuthology.orchestra.run.smithi187.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T23:01:47.380 INFO:teuthology.orchestra.run.smithi187.stdout:net.ipv4.ip_forward = 1 2022-03-08T23:01:47.439 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:01:47.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T23:01:47.483 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:01:47.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/docker/daemon.json 2022-03-08T23:01:47.495 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:01:47.496 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T23:01:47.512 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:01:47.513 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/docker/daemon.json 2022-03-08T23:01:47.569 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:01:47.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T23:01:47.585 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:01:47.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/docker/daemon.json 2022-03-08T23:01:47.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl restart docker || true 2022-03-08T23:01:47.646 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl restart docker || true 2022-03-08T23:01:47.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl restart docker || true 2022-03-08T23:01:48.739 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-08T23:01:48.745 INFO:tasks.kubeadm:172.21.15.107 is in 172.21.0.0/20 at pos 3946 2022-03-08T23:01:48.761 INFO:tasks.kubeadm:Pod subnet: 10.251.80.0/21 2022-03-08T23:01:48.761 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-08T23:01:48.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T23:01:48.764 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T23:01:48.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T23:01:48.791 INFO:teuthology.orchestra.run.smithi107.stderr: 2022-03-08T23:01:48.792 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:48.792 INFO:teuthology.orchestra.run.smithi107.stderr: 2022-03-08T23:01:48.795 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-03-08T23:01:48.795 INFO:teuthology.orchestra.run.smithi155.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:48.795 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-03-08T23:01:48.799 INFO:teuthology.orchestra.run.smithi187.stderr: 2022-03-08T23:01:48.799 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:48.799 INFO:teuthology.orchestra.run.smithi187.stderr: 2022-03-08T23:01:48.913 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T23:01:49.057 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T23:01:49.057 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T23:01:49.060 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T23:01:49.068 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T23:01:49.068 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T23:01:49.130 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T23:01:49.140 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T23:01:49.149 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T23:01:49.241 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T23:01:49.254 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T23:01:49.259 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T23:01:50.413 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-03-08T23:01:50.533 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2022-03-08T23:01:50.559 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2022-03-08T23:01:50.578 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-03-08T23:01:50.579 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-03-08T23:01:50.672 INFO:teuthology.orchestra.run.smithi155.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T23:01:50.686 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-03-08T23:01:50.686 INFO:teuthology.orchestra.run.smithi155.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:50.686 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-03-08T23:01:50.708 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2022-03-08T23:01:50.708 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2022-03-08T23:01:50.733 INFO:teuthology.orchestra.run.smithi187.stdout:Building dependency tree... 2022-03-08T23:01:50.733 INFO:teuthology.orchestra.run.smithi187.stdout:Reading state information... 2022-03-08T23:01:50.737 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-03-08T23:01:50.796 INFO:teuthology.orchestra.run.smithi107.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T23:01:50.806 INFO:teuthology.orchestra.run.smithi107.stderr: 2022-03-08T23:01:50.807 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:50.807 INFO:teuthology.orchestra.run.smithi107.stderr: 2022-03-08T23:01:50.822 INFO:teuthology.orchestra.run.smithi187.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T23:01:50.839 INFO:teuthology.orchestra.run.smithi187.stderr: 2022-03-08T23:01:50.839 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:50.839 INFO:teuthology.orchestra.run.smithi187.stderr: 2022-03-08T23:01:50.860 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2022-03-08T23:01:50.898 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2022-03-08T23:01:50.911 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-03-08T23:01:50.911 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-03-08T23:01:51.016 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2022-03-08T23:01:51.016 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2022-03-08T23:01:51.068 INFO:teuthology.orchestra.run.smithi187.stdout:Building dependency tree... 2022-03-08T23:01:51.069 INFO:teuthology.orchestra.run.smithi187.stdout:Reading state information... 2022-03-08T23:01:51.085 INFO:teuthology.orchestra.run.smithi155.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T23:01:51.085 INFO:teuthology.orchestra.run.smithi155.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T23:01:51.085 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2022-03-08T23:01:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: apt-transport-https 2022-03-08T23:01:51.179 INFO:teuthology.orchestra.run.smithi107.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T23:01:51.179 INFO:teuthology.orchestra.run.smithi107.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T23:01:51.180 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2022-03-08T23:01:51.180 INFO:teuthology.orchestra.run.smithi107.stdout: apt-transport-https 2022-03-08T23:01:51.230 INFO:teuthology.orchestra.run.smithi187.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T23:01:51.230 INFO:teuthology.orchestra.run.smithi187.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T23:01:51.230 INFO:teuthology.orchestra.run.smithi187.stdout:The following NEW packages will be installed: 2022-03-08T23:01:51.231 INFO:teuthology.orchestra.run.smithi187.stdout: apt-transport-https 2022-03-08T23:01:51.269 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T23:01:51.270 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 4,680 B of archives. 2022-03-08T23:01:51.270 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T23:01:51.270 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T23:01:51.363 INFO:teuthology.orchestra.run.smithi107.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T23:01:51.363 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 4,680 B of archives. 2022-03-08T23:01:51.363 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T23:01:51.364 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T23:01:51.411 INFO:teuthology.orchestra.run.smithi187.stdout:0 upgraded, 1 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T23:01:51.412 INFO:teuthology.orchestra.run.smithi187.stdout:Need to get 4,680 B of archives. 2022-03-08T23:01:51.412 INFO:teuthology.orchestra.run.smithi187.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T23:01:51.412 INFO:teuthology.orchestra.run.smithi187.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T23:01:51.490 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T23:01:51.491 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T23:01:51.491 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: falling back to frontend: Readline 2022-03-08T23:01:51.516 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T23:01:51.516 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T23:01:51.517 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: falling back to frontend: Teletype 2022-03-08T23:01:51.521 INFO:teuthology.orchestra.run.smithi155.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T23:01:51.569 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 4,680 B in 0s (26.6 kB/s) 2022-03-08T23:01:51.592 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T23:01:51.592 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T23:01:51.593 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: falling back to frontend: Readline 2022-03-08T23:01:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T23:01:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T23:01:51.623 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: falling back to frontend: Teletype 2022-03-08T23:01:51.629 INFO:teuthology.orchestra.run.smithi107.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T23:01:51.631 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T23:01:51.632 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T23:01:51.632 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: falling back to frontend: Readline 2022-03-08T23:01:51.648 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T23:01:51.650 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T23:01:51.651 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T23:01:51.651 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: falling back to frontend: Teletype 2022-03-08T23:01:51.656 INFO:teuthology.orchestra.run.smithi187.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T23:01:51.677 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 4,680 B in 0s (26.8 kB/s) 2022-03-08T23:01:51.691 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T23:01:51.694 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T23:01:51.705 INFO:teuthology.orchestra.run.smithi187.stdout:Fetched 4,680 B in 0s (26.7 kB/s) 2022-03-08T23:01:51.738 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T23:01:51.748 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T23:01:51.785 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T23:01:51.788 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T23:01:51.799 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T23:01:51.822 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T23:01:51.841 INFO:teuthology.orchestra.run.smithi187.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T23:01:51.845 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T23:01:51.881 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T23:01:52.007 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T23:01:52.133 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T23:01:52.142 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T23:01:52.500 INFO:teuthology.orchestra.run.smithi155.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T23:01:52.526 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-03-08T23:01:52.526 INFO:teuthology.orchestra.run.smithi155.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:52.526 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-03-08T23:01:52.625 INFO:teuthology.orchestra.run.smithi187.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T23:01:52.639 INFO:teuthology.orchestra.run.smithi187.stderr: 2022-03-08T23:01:52.639 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:52.640 INFO:teuthology.orchestra.run.smithi187.stderr: 2022-03-08T23:01:52.643 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T23:01:52.673 INFO:teuthology.orchestra.run.smithi107.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T23:01:52.687 INFO:teuthology.orchestra.run.smithi107.stderr: 2022-03-08T23:01:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:52.687 INFO:teuthology.orchestra.run.smithi107.stderr: 2022-03-08T23:01:52.756 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T23:01:52.765 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T23:01:52.817 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T23:01:52.838 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T23:01:52.878 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T23:01:52.921 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T23:01:52.928 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T23:01:52.961 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T23:01:53.011 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T23:01:53.048 INFO:teuthology.orchestra.run.smithi187.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T23:01:53.078 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T23:01:53.079 INFO:teuthology.orchestra.run.smithi187.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T23:01:53.079 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T23:01:53.094 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T23:01:53.281 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T23:01:53.316 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T23:01:53.411 INFO:teuthology.orchestra.run.smithi187.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T23:01:53.415 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 63.3 kB in 1s (83.8 kB/s) 2022-03-08T23:01:53.497 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 63.3 kB in 1s (94.4 kB/s) 2022-03-08T23:01:53.542 INFO:teuthology.orchestra.run.smithi187.stdout:Fetched 63.3 kB in 1s (82.9 kB/s) 2022-03-08T23:01:54.268 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-03-08T23:01:54.351 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2022-03-08T23:01:54.384 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2022-03-08T23:01:54.446 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-03-08T23:01:54.447 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-03-08T23:01:54.528 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2022-03-08T23:01:54.529 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2022-03-08T23:01:54.542 INFO:teuthology.orchestra.run.smithi155.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T23:01:54.555 INFO:teuthology.orchestra.run.smithi187.stdout:Building dependency tree... 2022-03-08T23:01:54.555 INFO:teuthology.orchestra.run.smithi187.stdout:Reading state information... 2022-03-08T23:01:54.559 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-03-08T23:01:54.560 INFO:teuthology.orchestra.run.smithi155.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:54.560 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-03-08T23:01:54.611 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-03-08T23:01:54.617 INFO:teuthology.orchestra.run.smithi107.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T23:01:54.629 INFO:teuthology.orchestra.run.smithi107.stderr: 2022-03-08T23:01:54.629 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:54.630 INFO:teuthology.orchestra.run.smithi107.stderr: 2022-03-08T23:01:54.645 INFO:teuthology.orchestra.run.smithi187.stdout:85 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T23:01:54.657 INFO:teuthology.orchestra.run.smithi187.stderr: 2022-03-08T23:01:54.658 INFO:teuthology.orchestra.run.smithi187.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T23:01:54.658 INFO:teuthology.orchestra.run.smithi187.stderr: 2022-03-08T23:01:54.683 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2022-03-08T23:01:54.712 INFO:teuthology.orchestra.run.smithi187.stdout:Reading package lists... 2022-03-08T23:01:54.786 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-03-08T23:01:54.787 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-03-08T23:01:54.856 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2022-03-08T23:01:54.857 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2022-03-08T23:01:54.887 INFO:teuthology.orchestra.run.smithi187.stdout:Building dependency tree... 2022-03-08T23:01:54.888 INFO:teuthology.orchestra.run.smithi187.stdout:Reading state information... 2022-03-08T23:01:54.930 INFO:teuthology.orchestra.run.smithi155.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T23:01:54.930 INFO:teuthology.orchestra.run.smithi155.stdout:bridge-utils set to manually installed. 2022-03-08T23:01:54.931 INFO:teuthology.orchestra.run.smithi155.stdout:The following additional packages will be installed: 2022-03-08T23:01:54.931 INFO:teuthology.orchestra.run.smithi155.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T23:01:54.932 INFO:teuthology.orchestra.run.smithi155.stdout:Suggested packages: 2022-03-08T23:01:54.932 INFO:teuthology.orchestra.run.smithi155.stdout: nftables 2022-03-08T23:01:54.978 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2022-03-08T23:01:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T23:01:54.993 INFO:teuthology.orchestra.run.smithi107.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T23:01:54.993 INFO:teuthology.orchestra.run.smithi107.stdout:bridge-utils set to manually installed. 2022-03-08T23:01:54.993 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2022-03-08T23:01:54.994 INFO:teuthology.orchestra.run.smithi107.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T23:01:54.995 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2022-03-08T23:01:54.995 INFO:teuthology.orchestra.run.smithi107.stdout: nftables 2022-03-08T23:01:55.039 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2022-03-08T23:01:55.040 INFO:teuthology.orchestra.run.smithi187.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T23:01:55.040 INFO:teuthology.orchestra.run.smithi187.stdout:bridge-utils set to manually installed. 2022-03-08T23:01:55.040 INFO:teuthology.orchestra.run.smithi187.stdout:The following additional packages will be installed: 2022-03-08T23:01:55.041 INFO:teuthology.orchestra.run.smithi107.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T23:01:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T23:01:55.042 INFO:teuthology.orchestra.run.smithi187.stdout:Suggested packages: 2022-03-08T23:01:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: nftables 2022-03-08T23:01:55.089 INFO:teuthology.orchestra.run.smithi187.stdout:The following NEW packages will be installed: 2022-03-08T23:01:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T23:01:55.171 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T23:01:55.172 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 73.6 MB of archives. 2022-03-08T23:01:55.172 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T23:01:55.172 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T23:01:55.233 INFO:teuthology.orchestra.run.smithi107.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T23:01:55.234 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 73.6 MB of archives. 2022-03-08T23:01:55.234 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T23:01:55.234 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T23:01:55.238 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T23:01:55.281 INFO:teuthology.orchestra.run.smithi187.stdout:0 upgraded, 8 newly installed, 0 to remove and 85 not upgraded. 2022-03-08T23:01:55.281 INFO:teuthology.orchestra.run.smithi187.stdout:Need to get 73.6 MB of archives. 2022-03-08T23:01:55.282 INFO:teuthology.orchestra.run.smithi187.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T23:01:55.282 INFO:teuthology.orchestra.run.smithi187.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T23:01:55.310 INFO:teuthology.orchestra.run.smithi187.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T23:01:55.326 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T23:01:55.346 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T23:01:55.428 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T23:01:55.476 INFO:teuthology.orchestra.run.smithi187.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T23:01:55.565 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T23:01:55.573 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T23:01:55.575 INFO:teuthology.orchestra.run.smithi187.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T23:01:55.614 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T23:01:55.863 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T23:01:55.920 INFO:teuthology.orchestra.run.smithi187.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T23:01:56.414 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T23:01:56.561 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T23:01:56.902 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T23:01:57.001 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T23:01:57.070 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T23:01:57.181 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T23:01:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T23:01:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T23:01:57.479 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: falling back to frontend: Readline 2022-03-08T23:01:57.488 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T23:01:57.488 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T23:01:57.489 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: falling back to frontend: Teletype 2022-03-08T23:01:57.494 INFO:teuthology.orchestra.run.smithi155.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T23:01:57.543 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 73.6 MB in 2s (32.4 MB/s) 2022-03-08T23:01:57.558 INFO:teuthology.orchestra.run.smithi187.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T23:01:57.578 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T23:01:57.578 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T23:01:57.578 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: falling back to frontend: Readline 2022-03-08T23:01:57.586 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T23:01:57.586 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T23:01:57.586 INFO:teuthology.orchestra.run.smithi107.stderr:debconf: falling back to frontend: Teletype 2022-03-08T23:01:57.591 INFO:teuthology.orchestra.run.smithi107.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T23:01:57.643 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 73.6 MB in 2s (31.7 MB/s) 2022-03-08T23:01:57.663 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package conntrack. 2022-03-08T23:01:57.707 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T23:01:57.711 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T23:01:57.729 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package conntrack. 2022-03-08T23:01:57.753 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T23:01:57.771 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T23:01:57.774 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T23:01:57.819 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T23:01:58.006 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package cri-tools. 2022-03-08T23:01:58.018 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T23:01:58.047 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T23:01:58.113 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cri-tools. 2022-03-08T23:01:58.127 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T23:01:58.129 INFO:teuthology.orchestra.run.smithi187.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T23:01:58.163 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T23:01:58.311 INFO:teuthology.orchestra.run.smithi187.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T23:01:58.714 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T23:01:58.715 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T23:01:58.715 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: falling back to frontend: Readline 2022-03-08T23:01:58.722 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T23:01:58.722 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T23:01:58.722 INFO:teuthology.orchestra.run.smithi187.stderr:debconf: falling back to frontend: Teletype 2022-03-08T23:01:58.727 INFO:teuthology.orchestra.run.smithi187.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T23:01:58.777 INFO:teuthology.orchestra.run.smithi187.stdout:Fetched 73.6 MB in 3s (21.7 MB/s) 2022-03-08T23:01:58.855 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package conntrack. 2022-03-08T23:01:58.901 INFO:teuthology.orchestra.run.smithi187.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T23:01:58.905 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T23:01:58.954 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T23:01:59.042 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ebtables. 2022-03-08T23:01:59.056 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T23:01:59.107 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T23:01:59.207 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package cri-tools. 2022-03-08T23:01:59.222 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T23:01:59.240 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ebtables. 2022-03-08T23:01:59.255 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T23:01:59.265 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T23:01:59.315 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T23:01:59.379 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T23:01:59.392 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T23:01:59.429 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T23:01:59.619 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T23:01:59.632 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T23:01:59.669 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T23:02:00.318 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package ebtables. 2022-03-08T23:02:00.332 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T23:02:00.401 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T23:02:00.654 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T23:02:00.667 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T23:02:00.704 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T23:02:00.850 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package socat. 2022-03-08T23:02:00.861 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T23:02:00.898 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T23:02:01.104 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package socat. 2022-03-08T23:02:01.117 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T23:02:01.154 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T23:02:01.305 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package kubelet. 2022-03-08T23:02:01.318 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T23:02:01.363 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T23:02:01.591 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package kubelet. 2022-03-08T23:02:01.604 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T23:02:01.641 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T23:02:02.240 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package socat. 2022-03-08T23:02:02.253 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T23:02:02.290 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T23:02:02.688 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package kubelet. 2022-03-08T23:02:02.700 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T23:02:02.737 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T23:02:03.732 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package kubectl. 2022-03-08T23:02:03.744 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T23:02:03.781 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T23:02:04.068 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package kubectl. 2022-03-08T23:02:04.081 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T23:02:04.145 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T23:02:04.950 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package kubeadm. 2022-03-08T23:02:04.964 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T23:02:04.992 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T23:02:05.123 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package kubectl. 2022-03-08T23:02:05.136 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T23:02:05.173 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T23:02:05.381 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package kubeadm. 2022-03-08T23:02:05.393 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T23:02:05.431 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T23:02:06.124 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T23:02:06.248 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T23:02:06.326 INFO:teuthology.orchestra.run.smithi187.stdout:Selecting previously unselected package kubeadm. 2022-03-08T23:02:06.339 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T23:02:06.375 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T23:02:06.376 INFO:teuthology.orchestra.run.smithi187.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T23:02:06.588 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T23:02:06.723 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T23:02:06.855 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T23:02:07.164 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T23:02:07.289 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T23:02:07.415 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T23:02:07.490 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T23:02:07.541 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T23:02:07.590 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T23:02:07.718 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T23:02:07.719 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T23:02:07.761 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T23:02:07.862 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T23:02:07.988 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T23:02:08.147 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T23:02:08.267 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T23:02:08.372 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T23:02:08.539 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T23:02:08.665 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T23:02:08.791 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T23:02:08.803 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T23:02:08.882 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T23:02:08.917 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T23:02:09.140 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T23:02:09.267 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T23:02:09.643 INFO:teuthology.orchestra.run.smithi187.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T23:02:10.045 INFO:teuthology.orchestra.run.smithi187.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T23:02:10.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T23:02:10.911 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T23:02:10.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T23:02:23.065 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T23:02:23.773 INFO:teuthology.orchestra.run.smithi107.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T23:02:24.562 INFO:teuthology.orchestra.run.smithi187.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T23:02:25.991 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T23:02:27.100 INFO:teuthology.orchestra.run.smithi107.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T23:02:27.539 INFO:teuthology.orchestra.run.smithi187.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T23:02:28.256 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T23:02:29.380 INFO:teuthology.orchestra.run.smithi107.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T23:02:29.952 INFO:teuthology.orchestra.run.smithi187.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T23:02:31.706 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T23:02:33.172 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T23:02:33.263 INFO:teuthology.orchestra.run.smithi107.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T23:02:33.727 INFO:teuthology.orchestra.run.smithi187.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T23:02:34.696 INFO:teuthology.orchestra.run.smithi107.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T23:02:35.366 INFO:teuthology.orchestra.run.smithi187.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T23:02:38.421 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T23:02:40.504 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T23:02:40.811 INFO:teuthology.orchestra.run.smithi107.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T23:02:41.142 INFO:teuthology.orchestra.run.smithi187.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T23:02:42.836 INFO:teuthology.orchestra.run.smithi107.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T23:02:43.185 INFO:teuthology.orchestra.run.smithi187.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T23:02:43.188 INFO:tasks.kubeadm:Token: abcdef.q74y6v5ymeq6cx4o 2022-03-08T23:02:43.188 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): '172.21.15.107', Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): '172.21.15.155', Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): '172.21.15.187'} 2022-03-08T23:02:43.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo kubeadm init --node-name smithi107 --token abcdef.q74y6v5ymeq6cx4o --pod-network-cidr 10.251.80.0/21 2022-03-08T23:02:43.394 INFO:teuthology.orchestra.run.smithi107.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-08T23:02:43.395 INFO:teuthology.orchestra.run.smithi107.stdout:[preflight] Running pre-flight checks 2022-03-08T23:02:43.785 INFO:teuthology.orchestra.run.smithi107.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-08T23:02:43.786 INFO:teuthology.orchestra.run.smithi107.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-08T23:02:43.786 INFO:teuthology.orchestra.run.smithi107.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-08T23:02:44.093 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-08T23:02:44.300 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "ca" certificate and key 2022-03-08T23:02:44.415 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "apiserver" certificate and key 2022-03-08T23:02:44.415 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi107] and IPs [10.96.0.1 172.21.15.107] 2022-03-08T23:02:44.612 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-08T23:02:44.853 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-08T23:02:45.051 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-08T23:02:45.307 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-08T23:02:45.632 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-08T23:02:45.632 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi107] and IPs [172.21.15.107 127.0.0.1 ::1] 2022-03-08T23:02:45.783 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-08T23:02:45.783 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi107] and IPs [172.21.15.107 127.0.0.1 ::1] 2022-03-08T23:02:45.913 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-08T23:02:46.209 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-08T23:02:46.322 INFO:teuthology.orchestra.run.smithi107.stdout:[certs] Generating "sa" key and public key 2022-03-08T23:02:46.322 INFO:teuthology.orchestra.run.smithi107.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-08T23:02:46.373 INFO:teuthology.orchestra.run.smithi107.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-08T23:02:46.554 INFO:teuthology.orchestra.run.smithi107.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-08T23:02:46.656 INFO:teuthology.orchestra.run.smithi107.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-08T23:02:46.818 INFO:teuthology.orchestra.run.smithi107.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-08T23:02:46.833 INFO:teuthology.orchestra.run.smithi107.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T23:02:46.834 INFO:teuthology.orchestra.run.smithi107.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T23:02:46.835 INFO:teuthology.orchestra.run.smithi107.stdout:[kubelet-start] Starting the kubelet 2022-03-08T23:02:47.045 INFO:teuthology.orchestra.run.smithi107.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-08T23:02:47.045 INFO:teuthology.orchestra.run.smithi107.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-08T23:02:47.047 INFO:teuthology.orchestra.run.smithi107.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-08T23:02:47.048 INFO:teuthology.orchestra.run.smithi107.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-08T23:02:47.050 INFO:teuthology.orchestra.run.smithi107.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-08T23:02:47.051 INFO:teuthology.orchestra.run.smithi107.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-08T23:03:05.055 INFO:teuthology.orchestra.run.smithi107.stdout:[apiclient] All control plane components are healthy after 18.003061 seconds 2022-03-08T23:03:05.055 INFO:teuthology.orchestra.run.smithi107.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-08T23:03:05.078 INFO:teuthology.orchestra.run.smithi107.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-08T23:03:05.078 INFO:teuthology.orchestra.run.smithi107.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-08T23:03:05.699 INFO:teuthology.orchestra.run.smithi107.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-08T23:03:05.699 INFO:teuthology.orchestra.run.smithi107.stdout:[mark-control-plane] Marking the node smithi107 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-08T23:03:05.700 INFO:teuthology.orchestra.run.smithi107.stdout:[mark-control-plane] Marking the node smithi107 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-08T23:03:06.208 INFO:teuthology.orchestra.run.smithi107.stdout:[bootstrap-token] Using token: abcdef.q74y6v5ymeq6cx4o 2022-03-08T23:03:06.208 INFO:teuthology.orchestra.run.smithi107.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-08T23:03:06.243 INFO:teuthology.orchestra.run.smithi107.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-08T23:03:06.269 INFO:teuthology.orchestra.run.smithi107.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-08T23:03:06.326 INFO:teuthology.orchestra.run.smithi107.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-08T23:03:06.331 INFO:teuthology.orchestra.run.smithi107.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-08T23:03:06.352 INFO:teuthology.orchestra.run.smithi107.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-08T23:03:06.394 INFO:teuthology.orchestra.run.smithi107.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-08T23:03:06.983 INFO:teuthology.orchestra.run.smithi107.stdout:[addons] Applied essential addon: CoreDNS 2022-03-08T23:03:07.147 INFO:teuthology.orchestra.run.smithi107.stdout:[addons] Applied essential addon: kube-proxy 2022-03-08T23:03:07.148 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:03:07.149 INFO:teuthology.orchestra.run.smithi107.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-08T23:03:07.149 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:03:07.149 INFO:teuthology.orchestra.run.smithi107.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-08T23:03:07.149 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:03:07.150 INFO:teuthology.orchestra.run.smithi107.stdout: mkdir -p $HOME/.kube 2022-03-08T23:03:07.150 INFO:teuthology.orchestra.run.smithi107.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-08T23:03:07.150 INFO:teuthology.orchestra.run.smithi107.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-08T23:03:07.150 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:03:07.151 INFO:teuthology.orchestra.run.smithi107.stdout:Alternatively, if you are the root user, you can run: 2022-03-08T23:03:07.151 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:03:07.151 INFO:teuthology.orchestra.run.smithi107.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-08T23:03:07.151 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:03:07.151 INFO:teuthology.orchestra.run.smithi107.stdout:You should now deploy a pod network to the cluster. 2022-03-08T23:03:07.152 INFO:teuthology.orchestra.run.smithi107.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-08T23:03:07.152 INFO:teuthology.orchestra.run.smithi107.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-08T23:03:07.152 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:03:07.152 INFO:teuthology.orchestra.run.smithi107.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-08T23:03:07.153 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:03:07.153 INFO:teuthology.orchestra.run.smithi107.stdout:kubeadm join 172.21.15.107:6443 --token abcdef.q74y6v5ymeq6cx4o \ 2022-03-08T23:03:07.153 INFO:teuthology.orchestra.run.smithi107.stdout: --discovery-token-ca-cert-hash sha256:a22df065f278aaf574a9e01dc5fa3719df3f49c20338a58423e0aa4208c95207 2022-03-08T23:03:07.154 DEBUG:teuthology.orchestra.run.smithi155:> sudo kubeadm join 172.21.15.107:6443 --node-name smithi155 --token abcdef.q74y6v5ymeq6cx4o --discovery-token-unsafe-skip-ca-verification 2022-03-08T23:03:07.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo kubeadm join 172.21.15.107:6443 --node-name smithi187 --token abcdef.q74y6v5ymeq6cx4o --discovery-token-unsafe-skip-ca-verification 2022-03-08T23:03:07.198 INFO:teuthology.orchestra.run.smithi155.stdout:[preflight] Running pre-flight checks 2022-03-08T23:03:07.200 INFO:teuthology.orchestra.run.smithi187.stdout:[preflight] Running pre-flight checks 2022-03-08T23:03:20.052 INFO:teuthology.orchestra.run.smithi155.stdout:[preflight] Reading configuration from the cluster... 2022-03-08T23:03:20.052 INFO:teuthology.orchestra.run.smithi155.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-08T23:03:20.456 INFO:teuthology.orchestra.run.smithi187.stdout:[preflight] Reading configuration from the cluster... 2022-03-08T23:03:20.457 INFO:teuthology.orchestra.run.smithi187.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-08T23:03:20.495 INFO:teuthology.orchestra.run.smithi155.stderr:W0308 23:03:20.497373 10742 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-08T23:03:20.557 INFO:teuthology.orchestra.run.smithi155.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T23:03:20.557 INFO:teuthology.orchestra.run.smithi155.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T23:03:20.557 INFO:teuthology.orchestra.run.smithi155.stdout:[kubelet-start] Starting the kubelet 2022-03-08T23:03:20.787 INFO:teuthology.orchestra.run.smithi155.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-08T23:03:21.133 INFO:teuthology.orchestra.run.smithi187.stderr:W0308 23:03:21.145200 10741 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-08T23:03:21.283 INFO:teuthology.orchestra.run.smithi187.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T23:03:21.284 INFO:teuthology.orchestra.run.smithi187.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T23:03:21.284 INFO:teuthology.orchestra.run.smithi187.stdout:[kubelet-start] Starting the kubelet 2022-03-08T23:03:21.502 INFO:teuthology.orchestra.run.smithi187.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-08T23:03:27.034 INFO:teuthology.orchestra.run.smithi187.stdout: 2022-03-08T23:03:27.034 INFO:teuthology.orchestra.run.smithi187.stdout:This node has joined the cluster: 2022-03-08T23:03:27.034 INFO:teuthology.orchestra.run.smithi187.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-08T23:03:27.034 INFO:teuthology.orchestra.run.smithi187.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-08T23:03:27.035 INFO:teuthology.orchestra.run.smithi187.stdout: 2022-03-08T23:03:27.035 INFO:teuthology.orchestra.run.smithi187.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-08T23:03:27.035 INFO:teuthology.orchestra.run.smithi187.stdout: 2022-03-08T23:03:34.308 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-03-08T23:03:34.308 INFO:teuthology.orchestra.run.smithi155.stdout:This node has joined the cluster: 2022-03-08T23:03:34.308 INFO:teuthology.orchestra.run.smithi155.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-08T23:03:34.309 INFO:teuthology.orchestra.run.smithi155.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-08T23:03:34.309 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-03-08T23:03:34.309 INFO:teuthology.orchestra.run.smithi155.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-08T23:03:34.309 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-03-08T23:03:34.315 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:03:34.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-08T23:03:34.329 INFO:tasks.kubeadm:Setting up kubectl 2022-03-08T23:03:34.330 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T23:03:34.342 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T23:03:34.359 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T23:03:34.378 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:03:34.378 DEBUG:teuthology.orchestra.run.smithi107:> dd of=.kube/config 2022-03-08T23:03:34.386 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:03:34.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/root/.kube/config 2022-03-08T23:03:34.443 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:03:34.444 DEBUG:teuthology.orchestra.run.smithi155:> dd of=.kube/config 2022-03-08T23:03:34.451 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:03:34.452 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/root/.kube/config 2022-03-08T23:03:34.464 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:03:34.465 DEBUG:teuthology.orchestra.run.smithi187:> dd of=.kube/config 2022-03-08T23:03:34.473 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:03:34.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/root/.kube/config 2022-03-08T23:03:34.485 DEBUG:teuthology.orchestra.run.smithi107:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-08T23:03:37.051 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-08T23:03:37.151 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-08T23:03:37.199 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-08T23:03:37.323 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-08T23:03:37.350 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-08T23:03:37.406 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-08T23:03:37.510 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-08T23:03:37.559 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-08T23:03:37.701 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-08T23:03:37.734 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-08T23:03:37.784 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-08T23:03:37.893 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-08T23:03:37.943 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-08T23:03:38.041 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-08T23:03:38.096 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-08T23:03:38.220 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-08T23:03:38.225 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-08T23:03:38.277 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-08T23:03:38.420 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-08T23:03:38.480 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-08T23:03:38.614 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-08T23:03:38.761 INFO:teuthology.orchestra.run.smithi107.stdout:namespace/tigera-operator created 2022-03-08T23:03:38.763 INFO:teuthology.orchestra.run.smithi107.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T23:03:38.786 INFO:teuthology.orchestra.run.smithi107.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-08T23:03:38.815 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/tigera-operator created 2022-03-08T23:03:38.855 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T23:03:38.904 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T23:03:38.919 INFO:teuthology.orchestra.run.smithi107.stdout:deployment.apps/tigera-operator created 2022-03-08T23:03:38.926 DEBUG:teuthology.orchestra.run.smithi107:> kubectl create -f - 2022-03-08T23:03:42.082 INFO:teuthology.orchestra.run.smithi107.stdout:installation.operator.tigera.io/default created 2022-03-08T23:03:42.088 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-08T23:03:42.088 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:03:42.089 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:03:42.094 INFO:tasks.kubeadm:Scratch devices on smithi107: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:03:42.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T23:03:42.118 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records in 2022-03-08T23:03:42.118 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2022-03-08T23:03:42.118 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139821 s, 750 MB/s 2022-03-08T23:03:42.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T23:03:42.190 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records in 2022-03-08T23:03:42.191 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2022-03-08T23:03:42.191 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129489 s, 810 MB/s 2022-03-08T23:03:42.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T23:03:42.226 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records in 2022-03-08T23:03:42.226 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2022-03-08T23:03:42.226 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143839 s, 729 MB/s 2022-03-08T23:03:42.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T23:03:42.256 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records in 2022-03-08T23:03:42.256 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2022-03-08T23:03:42.256 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0160213 s, 654 MB/s 2022-03-08T23:03:42.258 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:03:42.258 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:03:42.266 INFO:tasks.kubeadm:Scratch devices on smithi155: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:03:42.267 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T23:03:42.292 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records in 2022-03-08T23:03:42.293 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2022-03-08T23:03:42.293 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124638 s, 841 MB/s 2022-03-08T23:03:42.294 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T23:03:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records in 2022-03-08T23:03:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2022-03-08T23:03:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012774 s, 821 MB/s 2022-03-08T23:03:42.318 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T23:03:42.340 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records in 2022-03-08T23:03:42.341 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2022-03-08T23:03:42.341 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121384 s, 864 MB/s 2022-03-08T23:03:42.342 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T23:03:42.365 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records in 2022-03-08T23:03:42.366 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2022-03-08T23:03:42.366 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0147862 s, 709 MB/s 2022-03-08T23:03:42.367 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:03:42.368 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:03:42.375 INFO:tasks.kubeadm:Scratch devices on smithi187: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:03:42.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T23:03:42.445 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records in 2022-03-08T23:03:42.445 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2022-03-08T23:03:42.446 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015919 s, 659 MB/s 2022-03-08T23:03:42.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T23:03:42.476 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records in 2022-03-08T23:03:42.477 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2022-03-08T23:03:42.477 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0164703 s, 637 MB/s 2022-03-08T23:03:42.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T23:03:42.507 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records in 2022-03-08T23:03:42.507 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2022-03-08T23:03:42.508 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163495 s, 641 MB/s 2022-03-08T23:03:42.509 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T23:03:42.538 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records in 2022-03-08T23:03:42.538 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2022-03-08T23:03:42.539 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163576 s, 641 MB/s 2022-03-08T23:03:42.556 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-08T23:03:42.556 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi107-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi107 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi107-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi107 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi107-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi107 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi107-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi107 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi155-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi155 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi155-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi155 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi155-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi155 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi155-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi155 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi187-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi187 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi187-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi187 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi187-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi187 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi187-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi187 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-08T23:03:42.557 DEBUG:teuthology.orchestra.run.smithi107:> kubectl create -f - 2022-03-08T23:03:43.407 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi107-lv-1 created 2022-03-08T23:03:43.412 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi107-lv-2 created 2022-03-08T23:03:43.467 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi107-lv-3 created 2022-03-08T23:03:43.478 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi107-lv-4 created 2022-03-08T23:03:43.493 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi155-lv-1 created 2022-03-08T23:03:43.516 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi155-lv-2 created 2022-03-08T23:03:43.530 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi155-lv-3 created 2022-03-08T23:03:43.557 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi155-lv-4 created 2022-03-08T23:03:43.562 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi187-lv-1 created 2022-03-08T23:03:43.640 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi187-lv-2 created 2022-03-08T23:03:43.643 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi187-lv-3 created 2022-03-08T23:03:43.654 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi187-lv-4 created 2022-03-08T23:03:43.662 INFO:teuthology.orchestra.run.smithi107.stdout:storageclass.storage.k8s.io/scratch created 2022-03-08T23:03:43.665 DEBUG:teuthology.orchestra.run.smithi107:> kubectl taint node smithi107 node-role.kubernetes.io/master- || true 2022-03-08T23:03:43.736 INFO:teuthology.orchestra.run.smithi107.stdout:node/smithi107 untainted 2022-03-08T23:03:43.740 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-08T23:03:43.740 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-08T23:03:43.753 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-08T23:03:43.753 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:03:43.754 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:03:43.761 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:03:43.762 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T23:03:43.767 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T23:03:43.768 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:43.768 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-08T23:03:43.768 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:43.768 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2022-03-08 23:03:42.120359459 +0000 2022-03-08T23:03:43.769 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2022-03-08 23:03:42.120359459 +0000 2022-03-08T23:03:43.769 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2022-03-08 23:03:42.120359459 +0000 2022-03-08T23:03:43.769 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2022-03-08T23:03:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2022-03-08T23:03:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2022-03-08T23:03:43.780 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000608509 s, 841 kB/s 2022-03-08T23:03:43.784 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T23:03:43.789 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T23:03:43.789 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:43.790 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-03-08T23:03:43.790 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:43.790 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2022-03-08 23:03:42.196359458 +0000 2022-03-08T23:03:43.790 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2022-03-08 23:03:42.196359458 +0000 2022-03-08T23:03:43.791 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2022-03-08 23:03:42.196359458 +0000 2022-03-08T23:03:43.791 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2022-03-08T23:03:43.798 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2022-03-08T23:03:43.798 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2022-03-08T23:03:43.798 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000504112 s, 1.0 MB/s 2022-03-08T23:03:43.802 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T23:03:43.850 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T23:03:43.850 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:43.851 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-08T23:03:43.851 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:43.851 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2022-03-08 23:03:42.224359458 +0000 2022-03-08T23:03:43.851 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2022-03-08 23:03:42.224359458 +0000 2022-03-08T23:03:43.852 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2022-03-08 23:03:42.224359458 +0000 2022-03-08T23:03:43.852 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2022-03-08T23:03:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2022-03-08T23:03:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2022-03-08T23:03:43.860 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00053644 s, 954 kB/s 2022-03-08T23:03:43.865 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T23:03:43.870 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T23:03:43.871 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:43.871 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-08T23:03:43.871 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:43.871 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2022-03-08 23:03:42.260359458 +0000 2022-03-08T23:03:43.872 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2022-03-08 23:03:42.260359458 +0000 2022-03-08T23:03:43.872 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2022-03-08 23:03:42.260359458 +0000 2022-03-08T23:03:43.872 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2022-03-08T23:03:43.879 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2022-03-08T23:03:43.879 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2022-03-08T23:03:43.879 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000572231 s, 895 kB/s 2022-03-08T23:03:43.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T23:03:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:loop 2022-03-08T23:03:44.004 INFO:tasks.nvme_loop:Connecting nvme_loop smithi107:/dev/vg_nvme/lv_1... 2022-03-08T23:03:44.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T23:03:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:1 2022-03-08T23:03:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/vg_nvme/lv_11 2022-03-08T23:03:44.067 INFO:tasks.nvme_loop:Connecting nvme_loop smithi107:/dev/vg_nvme/lv_2... 2022-03-08T23:03:44.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T23:03:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:1 2022-03-08T23:03:44.116 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/vg_nvme/lv_21 2022-03-08T23:03:44.134 INFO:tasks.nvme_loop:Connecting nvme_loop smithi107:/dev/vg_nvme/lv_3... 2022-03-08T23:03:44.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T23:03:44.155 INFO:teuthology.orchestra.run.smithi107.stdout:1 2022-03-08T23:03:44.180 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/vg_nvme/lv_31 2022-03-08T23:03:44.202 INFO:tasks.nvme_loop:Connecting nvme_loop smithi107:/dev/vg_nvme/lv_4... 2022-03-08T23:03:44.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T23:03:44.269 INFO:teuthology.orchestra.run.smithi107.stdout:1 2022-03-08T23:03:44.294 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/vg_nvme/lv_41 2022-03-08T23:03:44.319 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:03:44.319 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:03:44.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo nvme list 2022-03-08T23:03:44.340 INFO:teuthology.orchestra.run.smithi107.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T23:03:44.340 INFO:teuthology.orchestra.run.smithi107.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T23:03:44.341 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme0n1 PHFT620400NK400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T23:03:44.341 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme1n1 d0096b65c9a43982 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:44.341 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme2n1 7a5d61686f3dffab Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:44.341 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme3n1 36230de599682176 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:44.342 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-08T23:03:45.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo nvme list 2022-03-08T23:03:45.359 INFO:teuthology.orchestra.run.smithi107.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T23:03:45.360 INFO:teuthology.orchestra.run.smithi107.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T23:03:45.360 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme0n1 PHFT620400NK400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T23:03:45.360 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme1n1 d0096b65c9a43982 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:45.361 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme2n1 7a5d61686f3dffab Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:45.361 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme3n1 36230de599682176 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:45.361 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme4n1 c8948e3f34c34fda Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:45.362 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T23:03:45.363 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:03:45.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/scratch_devs 2022-03-08T23:03:45.374 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:03:45.374 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:03:45.382 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:03:45.383 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T23:03:45.432 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T23:03:45.432 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:45.432 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-08T23:03:45.433 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:45.433 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2022-03-08 23:03:42.296736616 +0000 2022-03-08T23:03:45.433 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2022-03-08 23:03:42.292736616 +0000 2022-03-08T23:03:45.433 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2022-03-08 23:03:42.292736616 +0000 2022-03-08T23:03:45.434 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2022-03-08T23:03:45.441 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2022-03-08T23:03:45.441 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2022-03-08T23:03:45.442 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.0006362 s, 805 kB/s 2022-03-08T23:03:45.445 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T23:03:45.451 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T23:03:45.451 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:45.451 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-08T23:03:45.451 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:45.452 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2022-03-08 23:03:42.320736616 +0000 2022-03-08T23:03:45.452 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2022-03-08 23:03:42.320736616 +0000 2022-03-08T23:03:45.452 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2022-03-08 23:03:42.320736616 +0000 2022-03-08T23:03:45.452 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2022-03-08T23:03:45.458 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2022-03-08T23:03:45.459 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2022-03-08T23:03:45.459 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000549769 s, 931 kB/s 2022-03-08T23:03:45.462 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T23:03:45.468 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T23:03:45.469 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:45.469 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-08T23:03:45.469 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:45.469 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2022-03-08 23:03:42.344736615 +0000 2022-03-08T23:03:45.470 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2022-03-08 23:03:42.344736615 +0000 2022-03-08T23:03:45.470 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2022-03-08 23:03:42.344736615 +0000 2022-03-08T23:03:45.470 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2022-03-08T23:03:45.477 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2022-03-08T23:03:45.477 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2022-03-08T23:03:45.477 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000479149 s, 1.1 MB/s 2022-03-08T23:03:45.482 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T23:03:45.532 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T23:03:45.532 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:45.533 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-03-08T23:03:45.533 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:45.533 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2022-03-08 23:03:42.368736615 +0000 2022-03-08T23:03:45.533 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2022-03-08 23:03:42.368736615 +0000 2022-03-08T23:03:45.533 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2022-03-08 23:03:42.368736615 +0000 2022-03-08T23:03:45.534 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2022-03-08T23:03:45.543 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2022-03-08T23:03:45.543 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2022-03-08T23:03:45.543 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000525935 s, 974 kB/s 2022-03-08T23:03:45.548 DEBUG:teuthology.orchestra.run.smithi155:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T23:03:45.626 INFO:teuthology.orchestra.run.smithi155.stdout:loop 2022-03-08T23:03:45.628 INFO:tasks.nvme_loop:Connecting nvme_loop smithi155:/dev/vg_nvme/lv_1... 2022-03-08T23:03:45.628 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T23:03:45.650 INFO:teuthology.orchestra.run.smithi155.stdout:1 2022-03-08T23:03:45.678 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_11 2022-03-08T23:03:45.702 INFO:tasks.nvme_loop:Connecting nvme_loop smithi155:/dev/vg_nvme/lv_2... 2022-03-08T23:03:45.702 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T23:03:45.728 INFO:teuthology.orchestra.run.smithi155.stdout:1 2022-03-08T23:03:45.750 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_21 2022-03-08T23:03:45.773 INFO:tasks.nvme_loop:Connecting nvme_loop smithi155:/dev/vg_nvme/lv_3... 2022-03-08T23:03:45.774 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T23:03:45.838 INFO:teuthology.orchestra.run.smithi155.stdout:1 2022-03-08T23:03:45.862 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_31 2022-03-08T23:03:45.885 INFO:tasks.nvme_loop:Connecting nvme_loop smithi155:/dev/vg_nvme/lv_4... 2022-03-08T23:03:45.885 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T23:03:45.907 INFO:teuthology.orchestra.run.smithi155.stdout:1 2022-03-08T23:03:45.936 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_41 2022-03-08T23:03:45.960 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:03:45.960 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:03:45.967 DEBUG:teuthology.orchestra.run.smithi155:> sudo nvme list 2022-03-08T23:03:46.027 INFO:teuthology.orchestra.run.smithi155.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T23:03:46.027 INFO:teuthology.orchestra.run.smithi155.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T23:03:46.027 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme0n1 CVFT623300GW400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T23:03:46.028 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme1n1 283bafb6b3242f13 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:46.028 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme2n1 4927bf1b10f910e1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:46.028 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme3n1 356b02181db08426 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:46.028 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme4n1 13c8e36d22b3f70d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:46.029 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T23:03:46.030 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:03:46.030 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/scratch_devs 2022-03-08T23:03:46.040 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:03:46.041 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:03:46.049 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T23:03:46.049 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T23:03:46.104 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T23:03:46.105 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:46.105 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-08T23:03:46.105 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:46.105 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2022-03-08 23:03:42.460280628 +0000 2022-03-08T23:03:46.106 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2022-03-08 23:03:42.460280628 +0000 2022-03-08T23:03:46.106 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2022-03-08 23:03:42.460280628 +0000 2022-03-08T23:03:46.106 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2022-03-08T23:03:46.114 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2022-03-08T23:03:46.114 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2022-03-08T23:03:46.114 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000622183 s, 823 kB/s 2022-03-08T23:03:46.119 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T23:03:46.125 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T23:03:46.125 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:46.125 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-08T23:03:46.126 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:46.126 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2022-03-08 23:03:42.492280628 +0000 2022-03-08T23:03:46.126 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2022-03-08 23:03:42.492280628 +0000 2022-03-08T23:03:46.126 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2022-03-08 23:03:42.492280628 +0000 2022-03-08T23:03:46.127 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2022-03-08T23:03:46.134 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2022-03-08T23:03:46.134 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2022-03-08T23:03:46.134 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000553 s, 926 kB/s 2022-03-08T23:03:46.138 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T23:03:46.144 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T23:03:46.144 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:46.144 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-08T23:03:46.145 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:46.145 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2022-03-08 23:03:42.520280627 +0000 2022-03-08T23:03:46.145 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2022-03-08 23:03:42.520280627 +0000 2022-03-08T23:03:46.145 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2022-03-08 23:03:42.520280627 +0000 2022-03-08T23:03:46.146 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2022-03-08T23:03:46.152 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2022-03-08T23:03:46.153 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2022-03-08T23:03:46.153 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000533384 s, 960 kB/s 2022-03-08T23:03:46.156 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T23:03:46.162 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T23:03:46.163 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T23:03:46.163 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-08T23:03:46.163 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T23:03:46.163 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2022-03-08 23:03:42.552280626 +0000 2022-03-08T23:03:46.164 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2022-03-08 23:03:42.552280626 +0000 2022-03-08T23:03:46.164 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2022-03-08 23:03:42.552280626 +0000 2022-03-08T23:03:46.164 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2022-03-08T23:03:46.172 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2022-03-08T23:03:46.172 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2022-03-08T23:03:46.172 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000543134 s, 943 kB/s 2022-03-08T23:03:46.176 DEBUG:teuthology.orchestra.run.smithi187:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T23:03:46.307 INFO:teuthology.orchestra.run.smithi187.stdout:loop 2022-03-08T23:03:46.308 INFO:tasks.nvme_loop:Connecting nvme_loop smithi187:/dev/vg_nvme/lv_1... 2022-03-08T23:03:46.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T23:03:46.329 INFO:teuthology.orchestra.run.smithi187.stdout:1 2022-03-08T23:03:46.354 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/vg_nvme/lv_11 2022-03-08T23:03:46.376 INFO:tasks.nvme_loop:Connecting nvme_loop smithi187:/dev/vg_nvme/lv_2... 2022-03-08T23:03:46.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T23:03:46.395 INFO:teuthology.orchestra.run.smithi187.stdout:1 2022-03-08T23:03:46.417 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/vg_nvme/lv_21 2022-03-08T23:03:46.439 INFO:tasks.nvme_loop:Connecting nvme_loop smithi187:/dev/vg_nvme/lv_3... 2022-03-08T23:03:46.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T23:03:46.505 INFO:teuthology.orchestra.run.smithi187.stdout:1 2022-03-08T23:03:46.527 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/vg_nvme/lv_31 2022-03-08T23:03:46.550 INFO:tasks.nvme_loop:Connecting nvme_loop smithi187:/dev/vg_nvme/lv_4... 2022-03-08T23:03:46.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T23:03:46.572 INFO:teuthology.orchestra.run.smithi187.stdout:1 2022-03-08T23:03:46.597 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/vg_nvme/lv_41 2022-03-08T23:03:46.618 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:03:46.618 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:03:46.624 DEBUG:teuthology.orchestra.run.smithi187:> sudo nvme list 2022-03-08T23:03:46.679 INFO:teuthology.orchestra.run.smithi187.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T23:03:46.679 INFO:teuthology.orchestra.run.smithi187.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T23:03:46.679 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/nvme0n1 PHFT620500MW400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T23:03:46.680 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/nvme1n1 48e622e56888ff9b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:46.680 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/nvme2n1 34fee03658516bfb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:46.680 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/nvme3n1 8682db797f57bc45 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:46.680 INFO:teuthology.orchestra.run.smithi187.stdout:/dev/nvme4n1 699a9668ac364c9b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T23:03:46.681 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T23:03:46.682 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:03:46.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/scratch_devs 2022-03-08T23:03:46.694 INFO:teuthology.run_tasks:Running task rook... 2022-03-08T23:03:46.765 INFO:tasks.rook:Rook start 2022-03-08T23:03:46.765 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3', 'rook_image': 'rook/ceph:master'} 2022-03-08T23:03:46.766 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T23:03:46.766 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-08T23:03:46.767 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-08T23:03:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:Cloning into 'rook'... 2022-03-08T23:03:49.129 INFO:tasks.rook:/home/teuthworker 2022-03-08T23:03:49.129 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/sbin:/usr/sbin ip addr show 2022-03-08T23:03:49.183 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-08T23:03:49.183 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-08T23:03:49.184 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2022-03-08T23:03:49.184 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2022-03-08T23:03:49.184 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2022-03-08T23:03:49.184 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2022-03-08T23:03:49.185 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T23:03:49.185 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2022-03-08T23:03:49.185 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2022-03-08T23:03:49.185 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T23:03:49.185 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2022-03-08T23:03:49.186 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2022-03-08T23:03:49.186 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T23:03:49.186 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2022-03-08T23:03:49.186 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-08T23:03:49.187 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2022-03-08T23:03:49.187 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-08T23:03:49.187 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42857sec preferred_lft 42857sec 2022-03-08T23:03:49.187 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2022-03-08T23:03:49.187 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2022-03-08T23:03:49.188 INFO:teuthology.orchestra.run.smithi107.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-08T23:03:49.188 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 02:42:f8:32:c4:6d brd ff:ff:ff:ff:ff:ff 2022-03-08T23:03:49.188 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-08T23:03:49.188 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2022-03-08T23:03:49.189 DEBUG:teuthology.orchestra.run.smithi107:> which systemctl 2022-03-08T23:03:49.196 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/systemctl 2022-03-08T23:03:49.197 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-08T23:03:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T23:03:49.202 DEBUG:teuthology.orchestra.run.smithi107:> true 2022-03-08T23:03:49.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo lsb_release -is 2022-03-08T23:03:49.296 INFO:teuthology.orchestra.run.smithi107.stdout:Ubuntu 2022-03-08T23:03:49.301 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T23:03:49.302 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-08T23:03:49.313 DEBUG:teuthology.orchestra.remote:smithi107:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:03:49.326 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:03:49.327 DEBUG:teuthology.orchestra.run.smithi107:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-08T23:03:49.339 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-08T23:03:49.381 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:03:49.381 DEBUG:teuthology.orchestra.run.smithi107:> dd of=operator.yaml 2022-03-08T23:03:49.394 INFO:tasks.rook:Deploying operator 2022-03-08T23:03:49.402 DEBUG:teuthology.orchestra.remote:smithi107:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:03:49.427 DEBUG:teuthology.orchestra.remote:smithi107:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:03:49.442 DEBUG:teuthology.orchestra.run.smithi107:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-08T23:03:49.695 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-08T23:03:49.724 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-08T23:03:49.748 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-08T23:03:49.815 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-08T23:03:49.917 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-08T23:03:50.001 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-08T23:03:50.087 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-08T23:03:50.167 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-08T23:03:50.202 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-08T23:03:50.235 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-08T23:03:50.358 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-08T23:03:50.366 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-08T23:03:50.422 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-08T23:03:50.535 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-08T23:03:50.571 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-08T23:03:50.592 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-08T23:03:50.618 INFO:teuthology.orchestra.run.smithi107.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-08T23:03:50.702 INFO:teuthology.orchestra.run.smithi107.stdout:namespace/rook-ceph created 2022-03-08T23:03:50.725 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T23:03:50.776 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-08T23:03:50.800 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-08T23:03:50.861 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T23:03:50.897 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-08T23:03:50.917 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T23:03:50.943 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T23:03:51.017 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T23:03:51.022 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T23:03:51.026 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T23:03:51.034 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T23:03:51.043 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T23:03:51.052 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T23:03:51.075 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-08T23:03:51.110 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T23:03:51.149 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-08T23:03:51.151 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T23:03:51.160 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T23:03:51.184 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T23:03:51.194 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T23:03:51.201 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T23:03:51.210 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-08T23:03:51.218 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-08T23:03:51.226 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-08T23:03:51.282 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-08T23:03:51.285 INFO:teuthology.orchestra.run.smithi107.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-08T23:03:51.293 INFO:teuthology.orchestra.run.smithi107.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T23:03:51.293 INFO:teuthology.orchestra.run.smithi107.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-08T23:03:51.301 INFO:teuthology.orchestra.run.smithi107.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-08T23:03:51.310 INFO:teuthology.orchestra.run.smithi107.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T23:03:51.318 INFO:teuthology.orchestra.run.smithi107.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-08T23:03:51.342 INFO:teuthology.orchestra.run.smithi107.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T23:03:51.351 INFO:teuthology.orchestra.run.smithi107.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T23:03:51.360 INFO:teuthology.orchestra.run.smithi107.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T23:03:51.368 INFO:teuthology.orchestra.run.smithi107.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T23:03:51.377 INFO:teuthology.orchestra.run.smithi107.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T23:03:51.425 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-08T23:03:51.428 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-08T23:03:51.453 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-08T23:03:51.468 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T23:03:51.476 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T23:03:51.484 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-08T23:03:51.493 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-08T23:03:51.517 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T23:03:51.557 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-08T23:03:51.560 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T23:03:51.568 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T23:03:51.576 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-08T23:03:51.584 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T23:03:51.593 INFO:teuthology.orchestra.run.smithi107.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T23:03:51.626 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-08T23:03:51.635 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/rook-ceph-mgr created 2022-03-08T23:03:51.643 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/rook-ceph-osd created 2022-03-08T23:03:51.702 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-08T23:03:51.709 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/rook-ceph-system created 2022-03-08T23:03:51.735 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-08T23:03:51.762 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-08T23:03:51.868 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-08T23:03:51.870 INFO:teuthology.orchestra.run.smithi107.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-08T23:03:51.877 INFO:teuthology.orchestra.run.smithi107.stdout:configmap/rook-ceph-operator-config created 2022-03-08T23:03:51.902 INFO:teuthology.orchestra.run.smithi107.stdout:deployment.apps/rook-ceph-operator created 2022-03-08T23:03:51.909 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:03:52.034 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:03:52.035 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-operator-75dd789779-prwwx 0/1 Pending 0 1s 2022-03-08T23:04:02.038 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:04:02.098 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:04:02.098 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-operator-75dd789779-prwwx 0/1 Pending 0 11s 2022-03-08T23:04:12.101 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:04:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:04:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-operator-75dd789779-prwwx 0/1 Pending 0 21s 2022-03-08T23:04:22.164 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:04:22.220 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:04:22.221 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-operator-75dd789779-prwwx 0/1 ContainerCreating 0 31s 2022-03-08T23:04:32.224 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:04:32.285 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:04:32.286 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-operator-75dd789779-prwwx 0/1 ContainerCreating 0 41s 2022-03-08T23:04:42.289 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:04:42.356 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:04:42.356 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-operator-75dd789779-prwwx 0/1 ContainerCreating 0 51s 2022-03-08T23:04:52.358 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:04:52.408 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:04:52.408 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-operator-75dd789779-prwwx 0/1 ContainerCreating 0 61s 2022-03-08T23:05:02.411 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:05:02.627 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:02.628 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-operator-75dd789779-prwwx 0/1 ContainerCreating 0 71s 2022-03-08T23:05:12.631 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T23:05:12.694 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:12.694 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-operator-75dd789779-prwwx 1/1 Running 0 81s 2022-03-08T23:05:12.695 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-prwwx 2022-03-08T23:05:12.704 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:05:12.704 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:05:12.754 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:05:12.754 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:05:12.763 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:05:12.763 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T23:05:12.772 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-08T23:05:12.773 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-08T23:05:12.773 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-08T23:05:12.773 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-08T23:05:12.774 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-08T23:05:12.774 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-08T23:05:12.774 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-08T23:05:12.775 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-08T23:05:12.779 DEBUG:teuthology.orchestra.run.smithi107:> kubectl create -f - 2022-03-08T23:05:12.787 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.489322 I | rookcmd: starting Rook v1.8.0-alpha.0.381.g235781e28 with arguments '/usr/local/bin/rook ceph operator' 2022-03-08T23:05:12.788 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.489520 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-08T23:05:12.788 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.489527 I | cephcmd: starting Rook-Ceph operator 2022-03-08T23:05:12.788 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.627446 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-08T23:05:12.788 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.635564 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:05:12.789 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.635612 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:05:12.789 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.636270 I | operator: setting up schemes 2022-03-08T23:05:12.789 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.639789 I | operator: setting up the controller-runtime manager 2022-03-08T23:05:12.789 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.946743 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:05:12.790 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949473 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:05:12.790 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949509 I | ceph-cluster-controller: successfully started 2022-03-08T23:05:12.790 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949578 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:05:12.790 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949599 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:05:12.791 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949617 I | ceph-block-pool-controller: successfully started 2022-03-08T23:05:12.791 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949644 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:05:12.791 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949662 I | ceph-object-realm-controller: successfully started 2022-03-08T23:05:12.791 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949677 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:05:12.792 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949688 I | ceph-object-zone-controller: successfully started 2022-03-08T23:05:12.792 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949777 I | ceph-object-controller: successfully started 2022-03-08T23:05:12.792 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949808 I | ceph-file-controller: successfully started 2022-03-08T23:05:12.792 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949843 I | ceph-nfs-controller: successfully started 2022-03-08T23:05:12.792 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949869 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:05:12.793 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949891 I | ceph-client-controller: successfully started 2022-03-08T23:05:12.793 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949910 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:05:12.793 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949930 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:05:12.793 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949947 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:05:12.794 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949961 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:05:12.794 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949976 I | ceph-bucket-topic: successfully started 2022-03-08T23:05:12.794 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949987 I | ceph-bucket-notification: successfully started 2022-03-08T23:05:12.794 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.949998 I | ceph-bucket-notification: successfully started 2022-03-08T23:05:12.795 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.950008 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:05:12.795 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:09.950954 I | operator: starting the controller-runtime manager 2022-03-08T23:05:12.795 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.059776 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:05:12.795 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.059820 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:05:12.795 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.059844 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:05:12.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.062840 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-08T23:05:12.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.062864 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-08T23:05:12.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.064408 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:05:12.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.065220 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-08T23:05:12.797 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.071520 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-08T23:05:12.797 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.071537 I | ceph-csi: CSI CephFS driver disabled 2022-03-08T23:05:12.797 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.071545 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-08T23:05:12.798 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.073206 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-08T23:05:12.798 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:10.078258 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-08T23:05:13.810 INFO:teuthology.orchestra.run.smithi107.stdout:configmap/rook-config-override created 2022-03-08T23:05:13.814 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-08T23:05:13.814 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:05:13.815 DEBUG:teuthology.orchestra.run.smithi107:> dd of=cluster.yaml 2022-03-08T23:05:13.821 DEBUG:teuthology.orchestra.run.smithi107:> kubectl create -f cluster.yaml 2022-03-08T23:05:16.249 INFO:teuthology.orchestra.run.smithi107.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-08T23:05:16.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.255560 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-08T23:05:16.264 DEBUG:teuthology.orchestra.remote:smithi107:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:05:16.267 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.267364 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:05:16.275 DEBUG:teuthology.orchestra.run.smithi107:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-08T23:05:16.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.332828 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-08T23:05:16.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.332902 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:05:16.350 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349138 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:05:16.350 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349167 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:05:16.351 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349181 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:05:16.351 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349188 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:05:16.351 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349194 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:05:16.352 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349201 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:05:16.352 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349207 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:05:16.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349213 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:05:16.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349220 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:05:16.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349228 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:05:16.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349234 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:05:16.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349249 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:05:16.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349254 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:05:16.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349259 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:05:16.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349264 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:05:16.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349271 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:05:16.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349386 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:05:16.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.349402 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:05:16.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:05:16.350078 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:05:16.626 INFO:teuthology.orchestra.run.smithi107.stdout:deployment.apps/rook-ceph-tools created 2022-03-08T23:05:16.631 INFO:tasks.rook:Waiting for tools container to start 2022-03-08T23:05:16.632 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:05:16.737 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:16.737 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 Pending 0 0s 2022-03-08T23:05:21.740 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:05:21.797 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:21.798 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 5s 2022-03-08T23:05:26.800 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:05:26.872 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:26.873 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 10s 2022-03-08T23:05:31.876 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:05:31.937 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:31.937 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 15s 2022-03-08T23:05:36.940 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:05:37.001 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:37.001 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 21s 2022-03-08T23:05:42.004 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:05:42.071 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:42.071 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 26s 2022-03-08T23:05:47.074 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:05:47.139 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:47.139 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 31s 2022-03-08T23:05:52.143 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:05:52.203 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:52.203 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 36s 2022-03-08T23:05:57.205 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:05:57.271 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:05:57.271 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 41s 2022-03-08T23:06:02.274 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:02.336 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:02.337 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 46s 2022-03-08T23:06:07.339 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:07.412 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:07.412 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 51s 2022-03-08T23:06:12.414 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:12.477 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:12.478 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 56s 2022-03-08T23:06:17.479 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:17.539 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:17.540 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 61s 2022-03-08T23:06:22.077 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:22.078196 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:06:22.077 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:22.078217 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:06:22.078 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:22.078227 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:06:22.086 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:22.086787 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:06:22.218 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:22.218910 I | op-mon: start running mons 2022-03-08T23:06:22.302 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:22.303109 I | op-mon: creating mon secrets for a new cluster 2022-03-08T23:06:22.315 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:22.316613 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-08T23:06:22.318 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:22.319436 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T23:06:22.542 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:22.631 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 66s 2022-03-08T23:06:23.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:23.005577 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:06:23.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:23.005858 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:06:24.603 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:24.603887 I | op-mon: targeting the mon count 3 2022-03-08T23:06:24.617 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:24.617956 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-08T23:06:24.693 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:24.694042 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-08T23:06:24.703 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:24.704792 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-08T23:06:25.480 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:25.480987 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi187 2022-03-08T23:06:25.480 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:25.481019 I | op-mon: mon a assigned to node smithi187 2022-03-08T23:06:25.604 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:25.605289 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi107 2022-03-08T23:06:25.604 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:25.605322 I | op-mon: mon b assigned to node smithi107 2022-03-08T23:06:25.819 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:25.820416 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi155 2022-03-08T23:06:25.819 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:25.820438 I | op-mon: mon c assigned to node smithi155 2022-03-08T23:06:25.827 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:25.827954 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-08T23:06:25.832 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:25.833675 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-08T23:06:26.079 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:26.080790 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-08T23:06:26.085 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:26.086660 I | op-mon: creating mon a 2022-03-08T23:06:26.342 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:26.336047 I | op-mon: mon "a" endpoint is [v2:10.99.149.173:3300,v1:10.99.149.173:6789] 2022-03-08T23:06:26.658 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:26.659392 I | op-mon: parsing mon endpoints: 2022-03-08T23:06:26.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:26.659440 W | op-mon: ignoring invalid monitor 2022-03-08T23:06:26.664 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:26.659491 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:06:26.664 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:26.659521 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:06:26.664 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:26.663873 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:06:26.664 INFO:tasks.rook.operator.smithi107.stdout:I0308 23:06:26.663955 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:06:27.017 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:27.017245 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789"]}] data:a=10.99.149.173:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:-1] 2022-03-08T23:06:27.017 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:27.017635 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T23:06:27.018 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:27.018301 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T23:06:27.632 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:27.790 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:27.791 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 71s 2022-03-08T23:06:27.812 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:27.813887 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:06:27.813 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:27.814073 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:06:28.087 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:28.088363 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-08T23:06:28.278 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:28.278840 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-08T23:06:29.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:29.005049 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789"]}] data:a=10.99.149.173:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:0] 2022-03-08T23:06:29.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:29.005081 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:06:29.205 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:29.205968 I | op-mon: mon a is not yet running 2022-03-08T23:06:29.205 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:29.205996 I | op-mon: mons running: [] 2022-03-08T23:06:32.793 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:32.851 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:32.851 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 76s 2022-03-08T23:06:37.854 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:37.918 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:37.918 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 81s 2022-03-08T23:06:42.920 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:42.979 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:42.979 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 86s 2022-03-08T23:06:47.981 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:48.076 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:48.076 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 92s 2022-03-08T23:06:49.339 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:49.340949 I | op-mon: mons running: [a] 2022-03-08T23:06:49.565 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:49.566711 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:06:49.566 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:49.566743 I | op-mon: mons created: 1 2022-03-08T23:06:49.833 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:49.834674 I | op-mon: waiting for mon quorum with [a] 2022-03-08T23:06:49.841 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:49.843594 I | op-mon: mons running: [a] 2022-03-08T23:06:50.063 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:50.064924 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:06:50.064 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:50.064947 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:06:50.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:50.354920 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:06:50.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:50.354950 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:06:50.647 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:50.648791 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:06:50.648 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:50.648821 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:06:50.933 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:50.934033 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:06:50.933 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:50.934082 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:06:51.228 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:51.228851 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:06:51.228 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:51.228888 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:06:51.451 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:51.452218 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:06:51.451 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:51.452252 I | op-mon: creating mon b 2022-03-08T23:06:51.574 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:51.575407 I | op-mon: mon "a" endpoint is [v2:10.99.149.173:3300,v1:10.99.149.173:6789] 2022-03-08T23:06:51.623 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:51.624973 I | op-mon: mon "b" endpoint is [v2:10.100.13.26:3300,v1:10.100.13.26:6789] 2022-03-08T23:06:51.730 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:51.732039 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789","10.100.13.26:6789"]}] data:a=10.99.149.173:6789,b=10.100.13.26:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:0] 2022-03-08T23:06:51.770 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:51.771961 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:06:51.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:51.772269 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:06:52.090 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:52.091441 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:06:52.096 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:52.098492 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:06:52.097 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:52.098515 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T23:06:52.458 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:52.459677 I | op-mon: mons running: [a b] 2022-03-08T23:06:52.683 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:52.684568 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:06:52.762 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:52.763808 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-08T23:06:53.077 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:53.137 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:53.137 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 97s 2022-03-08T23:06:53.585 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:53.586403 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789","10.100.13.26:6789"]}] data:a=10.99.149.173:6789,b=10.100.13.26:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:1] 2022-03-08T23:06:53.585 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:53.586429 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T23:06:53.857 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:53.859150 I | op-mon: mons running: [a b] 2022-03-08T23:06:54.074 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:54.075708 I | op-mon: Monitors in quorum: [a] 2022-03-08T23:06:54.074 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:54.075729 I | op-mon: mons created: 2 2022-03-08T23:06:54.332 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:54.333633 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T23:06:54.342 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:54.344336 I | op-mon: mons running: [a b] 2022-03-08T23:06:54.572 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:54.573983 W | op-mon: failed to find initial monitor b in mon map 2022-03-08T23:06:54.573 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:54.574007 W | op-mon: monitor b is not in quorum list 2022-03-08T23:06:58.144 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:06:58.212 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:06:58.212 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 102s 2022-03-08T23:06:59.586 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:59.587456 I | op-mon: mons running: [a b] 2022-03-08T23:06:59.847 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:59.848914 W | op-mon: failed to find initial monitor b in mon map 2022-03-08T23:06:59.847 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:06:59.848938 W | op-mon: monitor b is not in quorum list 2022-03-08T23:07:03.215 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:07:03.555 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:07:03.556 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 107s 2022-03-08T23:07:05.502 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:05.503864 I | op-mon: mons running: [a b] 2022-03-08T23:07:05.709 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:05.711340 W | op-mon: failed to find initial monitor b in mon map 2022-03-08T23:07:05.710 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:05.711357 W | op-mon: monitor b is not in quorum list 2022-03-08T23:07:08.558 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:07:08.668 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:07:08.669 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 112s 2022-03-08T23:07:09.647 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649196 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:07:09.648 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649238 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:07:09.648 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649251 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:07:09.649 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649258 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:07:09.649 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649264 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:07:09.649 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649269 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:07:09.649 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649274 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:07:09.650 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649279 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:07:09.650 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649284 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:07:09.650 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649288 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:07:09.650 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649294 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:07:09.650 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649298 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:07:09.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649303 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:07:09.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649308 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:07:09.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649312 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:07:09.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649318 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:07:09.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649322 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:07:09.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649327 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:07:09.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649332 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:07:09.653 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649337 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:07:09.653 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649342 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:07:09.653 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649347 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:07:09.653 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649356 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:07:09.654 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649364 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:07:09.654 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.649369 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:07:09.654 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.654232 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:07:09.662 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.664016 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:07:09.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.664043 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:07:09.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.664048 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:07:09.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.664053 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:07:09.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.664057 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:07:09.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.664061 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:07:09.664 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.664065 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:07:09.741 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.742830 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:07:09.742 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.742856 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:07:09.742 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.742862 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:07:09.897 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.894048 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:07:09.960 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.952156 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:07:09.960 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.952185 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:07:09.960 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:09.952194 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:07:10.066 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:10.067663 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:07:10.066 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:10.067688 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:07:10.067 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:10.067695 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:07:10.154 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:10.154742 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:07:10.329 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:10.323708 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:07:10.422 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:10.423013 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:07:10.422 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:10.423039 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:07:10.462 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:10.464164 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:07:12.039 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:11.960258 I | op-mon: mon b is not yet running 2022-03-08T23:07:12.039 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:11.960284 I | op-mon: mons running: [a] 2022-03-08T23:07:13.671 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:07:13.732 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:07:13.732 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 117s 2022-03-08T23:07:18.734 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:07:20.543 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:20.544981 I | op-mon: mon b is not yet running 2022-03-08T23:07:20.543 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:20.545015 I | op-mon: mons running: [a] 2022-03-08T23:07:20.545 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:07:20.545 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 2m4s 2022-03-08T23:07:25.546 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:07:25.658 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:07:25.658 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 2m9s 2022-03-08T23:07:25.665 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:25.667529 I | op-mon: mon b is not yet running 2022-03-08T23:07:25.665 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:25.667554 I | op-mon: mons running: [a] 2022-03-08T23:07:30.660 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:07:30.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:30.697570 I | op-mon: mon b is not yet running 2022-03-08T23:07:30.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:30.697592 I | op-mon: mons running: [a] 2022-03-08T23:07:30.746 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:07:30.746 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 2m14s 2022-03-08T23:07:35.750 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:07:35.802 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:35.804498 I | op-mon: mons running: [a b] 2022-03-08T23:07:35.962 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:07:35.962 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 0/1 ContainerCreating 0 2m19s 2022-03-08T23:07:37.429 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:37.431928 I | op-mon: Monitors in quorum: [a b] 2022-03-08T23:07:37.430 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:37.431957 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:07:37.706 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:37.708285 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:07:37.706 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:37.708323 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:07:38.158 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:38.160003 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:07:38.158 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:38.160033 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:07:38.393 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:38.395164 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:07:38.393 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:38.395183 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:07:38.616 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:38.618248 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:07:38.616 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:38.618277 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:07:39.066 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:39.068416 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:07:39.067 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:39.068451 I | op-mon: creating mon c 2022-03-08T23:07:39.214 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:39.216316 I | op-mon: mon "a" endpoint is [v2:10.99.149.173:3300,v1:10.99.149.173:6789] 2022-03-08T23:07:39.500 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:39.502454 I | op-mon: mon "b" endpoint is [v2:10.100.13.26:3300,v1:10.100.13.26:6789] 2022-03-08T23:07:39.563 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:39.565357 I | op-mon: mon "c" endpoint is [v2:10.97.212.176:3300,v1:10.97.212.176:6789] 2022-03-08T23:07:39.644 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:39.646640 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789","10.100.13.26:6789","10.97.212.176:6789"]}] data:a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:1] 2022-03-08T23:07:40.070 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:40.072428 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:07:40.074 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:40.072738 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:07:40.532 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:40.534182 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-03-08T23:07:40.537 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:40.540007 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:07:40.546 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:40.548695 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:07:40.550 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:40.553479 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:07:40.561 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:40.563456 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:07:40.696 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:40.698401 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-08T23:07:40.964 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T23:07:41.027 INFO:teuthology.orchestra.run.smithi107.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T23:07:41.027 INFO:teuthology.orchestra.run.smithi107.stdout:rook-ceph-tools-d6d7c985c-zvbgc 1/1 Running 0 2m25s 2022-03-08T23:07:41.029 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-08T23:07:41.029 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:07:41.472 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:41.474743 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.212.176:6789","10.99.149.173:6789","10.100.13.26:6789"]}] data:a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:07:41.473 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:41.474785 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:07:41.536 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:07:41.536 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:07:41.537 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:07:41.537 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:07:41.537 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:07:41.537 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:07:41.538 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:07:41.538 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:07:41.538 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:07:41.538 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:07:41.538 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:07:41.539 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:07:41.539 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:07:41.539 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:07:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:07:42.074 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:42.076550 I | op-mon: mon c is not yet running 2022-03-08T23:07:42.074 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:42.076582 I | op-mon: mons running: [a b] 2022-03-08T23:07:42.375 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:42.374545 I | op-mon: Monitors in quorum: [a b] 2022-03-08T23:07:42.376 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:42.374572 I | op-mon: mons created: 3 2022-03-08T23:07:42.645 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:42.647836 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:07:42.673 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:42.675907 I | op-mon: mon c is not yet running 2022-03-08T23:07:42.674 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:42.675936 I | op-mon: mons running: [a b] 2022-03-08T23:07:47.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:47.697929 I | op-mon: mon c is not yet running 2022-03-08T23:07:47.696 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:47.697955 I | op-mon: mons running: [a b] 2022-03-08T23:07:51.542 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:07:52.013 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:07:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:07:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:07:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:07:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:07:52.014 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:07:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:07:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:07:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:07:52.015 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:07:52.016 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:07:52.016 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:07:52.016 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:07:52.016 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:07:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:07:52.716 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:52.718682 I | op-mon: mon c is not yet running 2022-03-08T23:07:52.717 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:52.718715 I | op-mon: mons running: [a b] 2022-03-08T23:07:57.737 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:57.740006 I | op-mon: mon c is not yet running 2022-03-08T23:07:57.738 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:07:57.740032 I | op-mon: mons running: [a b] 2022-03-08T23:08:02.021 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:08:02.561 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:08:02.561 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:08:02.562 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:08:02.562 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:08:02.562 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:08:02.562 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:08:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:08:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:08:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:08:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:08:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:08:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:08:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:08:02.565 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:08:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:08:02.756 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:02.759340 I | op-mon: mon c is not yet running 2022-03-08T23:08:02.757 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:02.759367 I | op-mon: mons running: [a b] 2022-03-08T23:08:08.392 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:08.394114 I | op-mon: mon c is not yet running 2022-03-08T23:08:08.392 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:08.394133 I | op-mon: mons running: [a b] 2022-03-08T23:08:12.569 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:08:13.101 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:08:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:08:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:08:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:08:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:08:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:08:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:08:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:08:13.104 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:08:13.104 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:08:13.104 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:08:13.104 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:08:13.104 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:08:13.105 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:08:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:08:13.411 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:13.412589 I | op-mon: mon c is not yet running 2022-03-08T23:08:13.411 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:13.412617 I | op-mon: mons running: [a b] 2022-03-08T23:08:19.613 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:19.616195 I | op-mon: mon c is not yet running 2022-03-08T23:08:19.613 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:19.616217 I | op-mon: mons running: [a b] 2022-03-08T23:08:23.108 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:08:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:08:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:08:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:08:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:08:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:08:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:08:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:08:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:08:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:08:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:08:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:08:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:08:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:08:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:08:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:08:25.125 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:25.128943 I | op-mon: mon c is not yet running 2022-03-08T23:08:25.126 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:25.128964 I | op-mon: mons running: [a b] 2022-03-08T23:08:31.284 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:31.287731 I | op-mon: mon c is not yet running 2022-03-08T23:08:31.285 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:31.287776 I | op-mon: mons running: [a b] 2022-03-08T23:08:35.053 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:08:35.581 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:08:35.581 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:08:35.582 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:08:35.582 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:08:35.582 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:08:35.583 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:08:35.583 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:08:35.583 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:08:35.583 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:08:35.584 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:08:35.584 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:08:35.584 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:08:35.584 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:08:35.586 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:08:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:08:36.335 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:36.338567 I | op-mon: mon c is not yet running 2022-03-08T23:08:36.335 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:36.338596 I | op-mon: mons running: [a b] 2022-03-08T23:08:41.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:41.359740 I | op-mon: mon c is not yet running 2022-03-08T23:08:41.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:41.359765 I | op-mon: mons running: [a b] 2022-03-08T23:08:45.591 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:08:46.048 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:08:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:08:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:08:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:08:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:08:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:08:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:08:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:08:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:08:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:08:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:08:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:08:46.051 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:08:46.052 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:08:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:08:46.405 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:46.408563 I | op-mon: mon c is not yet running 2022-03-08T23:08:46.405 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:46.408593 I | op-mon: mons running: [a b] 2022-03-08T23:08:51.442 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:51.445607 I | op-mon: mon c is not yet running 2022-03-08T23:08:51.442 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:51.445636 I | op-mon: mons running: [a b] 2022-03-08T23:08:56.058 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:08:56.488 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:56.491939 I | op-mon: mon c is not yet running 2022-03-08T23:08:56.488 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:08:56.491971 I | op-mon: mons running: [a b] 2022-03-08T23:08:56.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:08:56.551 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:08:56.551 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:08:56.551 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:08:56.552 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:08:56.552 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:08:56.552 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:08:56.552 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:08:56.552 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:08:56.553 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:08:56.553 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:08:56.553 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:08:56.553 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:08:56.556 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:08:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:09:01.508 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:01.511633 I | op-mon: mon c is not yet running 2022-03-08T23:09:01.508 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:01.511661 I | op-mon: mons running: [a b] 2022-03-08T23:09:06.527 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:06.530961 I | op-mon: mon c is not yet running 2022-03-08T23:09:06.527 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:06.530995 I | op-mon: mons running: [a b] 2022-03-08T23:09:06.561 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:09:07.043 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:09:07.044 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:09:07.044 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:09:07.044 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:09:07.045 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:09:07.045 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:09:07.045 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:09:07.045 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:09:07.045 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:09:07.046 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:09:07.046 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:09:07.046 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:09:07.046 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:09:07.048 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:09:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:09:11.547 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:11.551420 I | op-mon: mon c is not yet running 2022-03-08T23:09:11.548 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:11.551449 I | op-mon: mons running: [a b] 2022-03-08T23:09:16.579 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:16.582384 I | op-mon: mons running: [a b c] 2022-03-08T23:09:17.052 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:09:18.389 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:09:18.390 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:09:18.390 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:09:18.390 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:09:18.391 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:09:18.391 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:09:18.391 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:09:18.391 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:09:18.392 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:09:18.392 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:09:18.392 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:09:18.392 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:09:18.392 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:09:18.393 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:09:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:09:19.790 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:19.794042 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:09:19.790 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:19.794073 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:09:20.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.026677 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:09:20.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.026706 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:09:20.247 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.251219 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:09:20.247 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.251246 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:09:20.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.459035 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:09:20.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.459072 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:09:20.741 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.745323 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:09:20.741 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.745354 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:09:20.963 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.966894 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:09:20.963 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:20.966948 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:09:21.293 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:21.297635 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:09:21.636 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:21.640228 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:09:21.978 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:21.981780 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:09:22.388 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:22.392430 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:09:22.389 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:22.392451 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:09:22.814 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:22.818308 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:09:23.120 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:23.124356 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:09:23.121 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:23.124388 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:09:23.121 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:23.124398 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:09:23.479 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:23.482819 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:09:23.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:23.496919 I | op-mgr: start running mgr 2022-03-08T23:09:23.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:23.496946 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:09:24.178 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:24.182803 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi155": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:09:24.306 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:24.309834 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi155": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:09:28.396 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:09:29.024 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:09:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2022-03-08T23:09:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:pg stat 2022-03-08T23:09:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:pg getmap 2022-03-08T23:09:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T23:09:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T23:09:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_pools_json 2022-03-08T23:09:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-pool [...] 2022-03-08T23:09:29.026 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-primary [] [...] 2022-03-08T23:09:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls-by-osd [] [...] 2022-03-08T23:09:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:pg ls [] [...] 2022-03-08T23:09:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T23:09:29.027 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2022-03-08T23:09:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 22 2022-03-08T23:09:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T23:09:39.033 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:09:39.618 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:09:39.619 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-08T23:09:39.622 INFO:teuthology.orchestra.run.smithi107.stderr:command terminated with exit code 2 2022-03-08T23:09:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-08T23:09:45.020 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.024074 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-08T23:09:45.024 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.027991 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:09:45.126 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.128540 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:09:45.126 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.128636 I | op-mgr: successful modules: balancer 2022-03-08T23:09:45.133 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.137594 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:09:45.133 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.137688 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T23:09:45.133 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.137750 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:09:45.185 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.188293 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:09:45.187 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.192341 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T23:09:45.188 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.192361 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:09:45.188 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.192365 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:09:45.656 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.660920 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:09:45.657 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.660952 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:09:45.666 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.671235 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-08T23:09:45.667 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.671270 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-08T23:09:45.667 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.671283 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-08T23:09:45.736 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.741075 I | op-mgr: successful modules: dashboard 2022-03-08T23:09:45.739 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:45.744445 I | op-mgr: successful modules: prometheus 2022-03-08T23:09:46.788 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:46.792559 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:09:49.625 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status -f json 2022-03-08T23:09:52.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:09:52.953012 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi107": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:10:01.059 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:01.064188 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:10:01.120 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:10:01.120 INFO:teuthology.orchestra.run.smithi107.stdout:{"available": true, "backend": "rook"} 2022-03-08T23:10:01.145 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-08T23:10:01.146 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-08T23:10:01.794 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch apply osd --all-available-devices 2022-03-08T23:10:02.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.496675 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:10:02.494 INFO:tasks.rook.operator.smithi107.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T23:10:02.494 INFO:tasks.rook.operator.smithi107.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:10:02.494 INFO:tasks.rook.operator.smithi107.stdout:   ... // 3 identical fields 2022-03-08T23:10:02.495 INFO:tasks.rook.operator.smithi107.stdout:   Config: nil, 2022-03-08T23:10:02.495 INFO:tasks.rook.operator.smithi107.stdout:   Selection: {}, 2022-03-08T23:10:02.495 INFO:tasks.rook.operator.smithi107.stdout:-  StorageClassDeviceSets: nil, 2022-03-08T23:10:02.495 INFO:tasks.rook.operator.smithi107.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:10:02.496 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.496 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi107-lv-1", 2022-03-08T23:10:02.496 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.500 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.500 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.501 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.501 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi107-lv-2", 2022-03-08T23:10:02.501 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.501 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.501 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.502 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.502 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi107-lv-3", 2022-03-08T23:10:02.502 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.502 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.503 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.503 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.503 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi107-lv-4", 2022-03-08T23:10:02.503 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.504 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.504 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.504 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.504 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi155-lv-1", 2022-03-08T23:10:02.505 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.505 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.505 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.505 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.506 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi155-lv-2", 2022-03-08T23:10:02.506 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.506 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.506 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.506 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.507 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi155-lv-3", 2022-03-08T23:10:02.507 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.507 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.507 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.508 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.508 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi155-lv-4", 2022-03-08T23:10:02.508 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.508 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.509 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.509 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.509 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi187-lv-1", 2022-03-08T23:10:02.509 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.510 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.510 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.510 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.510 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi187-lv-2", 2022-03-08T23:10:02.510 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.511 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.511 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.511 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.511 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi187-lv-3", 2022-03-08T23:10:02.512 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.512 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.512 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.512 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:10:02.513 INFO:teuthology.orchestra.run.smithi107.stdout:['Success'] 2022-03-08T23:10:02.515 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi187-lv-4", 2022-03-08T23:10:02.515 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:10:02.515 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T23:10:02.515 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.516 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:10:02.516 INFO:tasks.rook.operator.smithi107.stdout:   }, 2022-03-08T23:10:02.516 INFO:tasks.rook.operator.smithi107.stdout:   Annotations: nil, 2022-03-08T23:10:02.516 INFO:tasks.rook.operator.smithi107.stdout:   Labels: nil, 2022-03-08T23:10:02.517 INFO:tasks.rook.operator.smithi107.stdout:   ... // 20 identical fields 2022-03-08T23:10:02.517 INFO:tasks.rook.operator.smithi107.stdout:  } 2022-03-08T23:10:02.517 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.496799 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:10:02.517 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.496950 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-08T23:10:02.518 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.497113 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-08T23:10:02.518 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.497160 I | operator: successfully started the controller-runtime manager 2022-03-08T23:10:02.518 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.497201 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-08T23:10:02.518 INFO:tasks.rook.operator.smithi107.stdout:I0308 23:10:02.497227 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:10:02.519 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-08T23:10:02.520 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:10:02.537 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.542078 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:10:02.537 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.542109 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:10:02.538 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.542157 I | operator: setting up schemes 2022-03-08T23:10:02.542 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.547266 I | operator: setting up the controller-runtime manager 2022-03-08T23:10:02.847 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.852248 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:10:02.852 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854247 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:10:02.852 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854274 I | ceph-cluster-controller: successfully started 2022-03-08T23:10:02.853 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854329 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:10:02.853 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854346 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:10:02.853 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854364 I | ceph-block-pool-controller: successfully started 2022-03-08T23:10:02.853 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854385 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:10:02.854 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854414 I | ceph-object-realm-controller: successfully started 2022-03-08T23:10:02.854 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854430 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:10:02.854 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854442 I | ceph-object-zone-controller: successfully started 2022-03-08T23:10:02.854 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854600 I | ceph-object-controller: successfully started 2022-03-08T23:10:02.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854638 I | ceph-file-controller: successfully started 2022-03-08T23:10:02.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854672 I | ceph-nfs-controller: successfully started 2022-03-08T23:10:02.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854699 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:10:02.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854727 I | ceph-client-controller: successfully started 2022-03-08T23:10:02.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854747 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:10:02.856 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854774 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:10:02.856 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854793 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:10:02.856 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854812 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:10:02.856 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854829 I | ceph-bucket-topic: successfully started 2022-03-08T23:10:02.857 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854843 I | ceph-bucket-notification: successfully started 2022-03-08T23:10:02.857 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854858 I | ceph-bucket-notification: successfully started 2022-03-08T23:10:02.857 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.854872 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:10:02.857 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.855560 I | operator: starting the controller-runtime manager 2022-03-08T23:10:02.953 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.957490 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:10:02.953 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.957536 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:10:02.953 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.957549 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:10:02.956 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.959268 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:10:02.961 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.963293 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:10:02.964 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.968809 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:10:02.971 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.971259 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:10:02.971 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.971448 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:10:02.972 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.971497 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:10:02.972 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.972937 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:10:02.972 INFO:tasks.rook.operator.smithi107.stdout:I0308 23:10:02.973012 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:10:02.983 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.977583 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:10:02.983 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983797 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:10:02.983 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983823 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:10:02.983 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983830 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:10:02.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983834 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:10:02.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983840 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:10:02.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983845 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:10:02.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983851 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:10:02.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983856 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:10:02.985 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983861 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:10:02.985 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983866 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:10:02.985 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983870 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:10:02.985 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983876 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:10:02.985 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983881 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:10:02.986 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983921 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:10:02.986 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983945 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:10:02.986 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.983968 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:10:02.986 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.986717 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:10:02.987 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.986742 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:10:02.987 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:02.988735 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:10:03.044 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:10:03.063 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:10:03.064 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:10:11.115 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119340 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:10:11.115 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119367 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:10:11.116 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119374 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:10:11.116 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119378 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:10:11.116 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119381 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:10:11.116 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119385 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:10:11.117 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119388 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:10:11.117 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119391 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:10:11.117 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119395 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:10:11.118 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119398 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:10:11.119 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119402 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:10:11.119 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119408 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:10:11.119 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119413 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:10:11.119 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119417 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:10:11.120 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119420 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:10:11.120 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119423 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:10:11.120 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119426 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:10:11.120 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119429 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:10:11.121 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119432 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:10:11.121 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119436 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:10:11.121 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119439 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:10:11.121 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119443 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:10:11.121 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119446 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:10:11.122 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119452 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:10:11.122 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.119457 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:10:11.239 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.243982 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:10:11.248 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.253253 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:10:11.248 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.253273 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:10:11.249 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.253277 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:10:11.249 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.253279 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:10:11.249 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.253282 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:10:11.249 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.253285 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:10:11.250 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.253287 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:10:11.316 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.320571 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:10:11.316 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.320695 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:10:11.316 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.320723 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:10:11.449 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.454373 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:10:11.453 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.454797 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:10:11.453 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.454814 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:10:11.453 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.454820 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:10:11.475 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.480667 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:10:11.479 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.481084 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:10:11.547 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.551917 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:10:11.751 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.756303 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:10:11.752 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.756320 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:10:11.752 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.756324 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:10:11.842 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.846530 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:10:11.842 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.846545 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:10:11.842 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.846549 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:10:11.891 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.896499 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:10:11.901 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:11.906450 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:10:12.200 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:12.205487 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:10:12.218 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:12.223221 I | op-mon: start running mons 2022-03-08T23:10:12.278 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:12.283336 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:10:12.279 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:12.283378 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:10:12.341 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:12.345745 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:10:12.341 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:12.346011 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:10:13.065 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:10:13.293 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:13.298331 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789","10.100.13.26:6789","10.97.212.176:6789"]}] data:c=10.97.212.176:6789,a=10.99.149.173:6789,b=10.100.13.26:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:10:13.575 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:10:13.595 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:10:13.595 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:10:13.891 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:13.896141 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:10:13.892 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:13.896424 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:10:15.498 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:15.503199 I | op-mon: targeting the mon count 3 2022-03-08T23:10:15.503 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:15.508359 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:10:15.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:15.860056 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:10:15.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:15.860085 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:10:16.198 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:16.202946 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:10:16.198 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:16.202965 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:10:16.531 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:16.536217 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:10:16.532 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:16.536257 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:10:16.864 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:16.868999 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:10:16.864 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:16.869025 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:10:17.197 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:17.201983 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:10:17.197 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:17.202019 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:10:17.441 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:17.446384 I | op-mon: mon "a" endpoint is [v2:10.99.149.173:3300,v1:10.99.149.173:6789] 2022-03-08T23:10:17.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:17.605157 I | op-mon: mon "b" endpoint is [v2:10.100.13.26:3300,v1:10.100.13.26:6789] 2022-03-08T23:10:17.894 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:17.899292 I | op-mon: mon "c" endpoint is [v2:10.97.212.176:3300,v1:10.97.212.176:6789] 2022-03-08T23:10:18.498 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:18.503691 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789","10.100.13.26:6789","10.97.212.176:6789"]}] data:a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:10:19.093 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:19.097807 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:10:19.093 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:19.098094 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:10:19.508 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:19.513557 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:10:19.522 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:19.526950 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:10:19.522 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:19.526981 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:20.098 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:20.103632 I | op-mon: mons running: [a b c] 2022-03-08T23:10:20.517 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:20.521995 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:20.523 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:20.528721 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:10:20.531 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:20.536534 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:10:20.531 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:20.536557 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:20.698 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:20.703268 I | op-mon: mons running: [a b c] 2022-03-08T23:10:21.102 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:21.107442 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:21.108 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:21.113886 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:10:21.118 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:21.123779 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:10:21.118 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:21.123798 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:21.297 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:21.302773 I | op-mon: mons running: [a b c] 2022-03-08T23:10:21.702 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:21.706893 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:21.702 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:21.706919 I | op-mon: mons created: 3 2022-03-08T23:10:22.105 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:22.109892 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:10:22.124 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:22.129020 I | op-mon: mons running: [a b c] 2022-03-08T23:10:22.519 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:22.524098 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:10:22.519 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:22.524143 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:10:22.990 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:22.995535 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:10:23.408 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:23.413273 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:10:23.597 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:10:23.918 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:23.923550 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:10:24.166 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:10:24.184 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:10:24.185 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:10:24.440 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:24.445713 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:10:24.441 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:24.445736 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:10:24.881 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:24.886791 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:10:25.287 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:25.291639 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:10:25.287 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:25.291676 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:10:25.287 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:25.291684 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:10:25.720 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:25.725580 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:10:25.746 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:25.751240 I | op-mgr: start running mgr 2022-03-08T23:10:25.746 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:25.751285 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:10:26.200 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.205837 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:10:26.209 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.214932 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:10:26.212 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.218012 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:10:26.382 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.387371 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:10:26.390 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.387528 I | op-mgr: successful modules: balancer 2022-03-08T23:10:26.399 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.404255 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:10:26.399 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.404289 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:10:26.419 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.424203 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:10:26.582 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.587898 I | op-osd: creating 1 new PVCs for device set "smithi107-lv-1" 2022-03-08T23:10:26.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.801788 I | op-osd: successfully provisioned PVC "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:10:26.797 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.801817 I | op-osd: creating 1 new PVCs for device set "smithi107-lv-2" 2022-03-08T23:10:26.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.989318 I | op-osd: successfully provisioned PVC "smithi107-lv-2-data-09kfs9" 2022-03-08T23:10:26.985 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:26.989359 I | op-osd: creating 1 new PVCs for device set "smithi107-lv-3" 2022-03-08T23:10:27.233 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.238379 I | op-mgr: successful modules: prometheus 2022-03-08T23:10:27.240 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.245793 I | op-mgr: successful modules: dashboard 2022-03-08T23:10:27.243 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.246571 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:10:27.268 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.273664 I | op-osd: successfully provisioned PVC "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:10:27.268 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.273696 I | op-osd: creating 1 new PVCs for device set "smithi107-lv-4" 2022-03-08T23:10:27.404 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.409476 I | op-osd: successfully provisioned PVC "smithi107-lv-4-data-0mptlv" 2022-03-08T23:10:27.404 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.409506 I | op-osd: creating 1 new PVCs for device set "smithi155-lv-1" 2022-03-08T23:10:27.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.588147 I | op-osd: successfully provisioned PVC "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:10:27.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.588201 I | op-osd: creating 1 new PVCs for device set "smithi155-lv-2" 2022-03-08T23:10:27.787 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.792650 I | op-osd: successfully provisioned PVC "smithi155-lv-2-data-0v2l2j" 2022-03-08T23:10:27.788 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.792683 I | op-osd: creating 1 new PVCs for device set "smithi155-lv-3" 2022-03-08T23:10:27.982 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.987225 I | op-osd: successfully provisioned PVC "smithi155-lv-3-data-08gx6j" 2022-03-08T23:10:27.982 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:27.987256 I | op-osd: creating 1 new PVCs for device set "smithi155-lv-4" 2022-03-08T23:10:28.184 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:28.189687 I | op-osd: successfully provisioned PVC "smithi155-lv-4-data-04c4td" 2022-03-08T23:10:28.185 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:28.189739 I | op-osd: creating 1 new PVCs for device set "smithi187-lv-1" 2022-03-08T23:10:28.383 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:28.388199 I | op-osd: successfully provisioned PVC "smithi187-lv-1-data-067hpz" 2022-03-08T23:10:28.383 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:28.388244 I | op-osd: creating 1 new PVCs for device set "smithi187-lv-2" 2022-03-08T23:10:28.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:28.588237 I | op-osd: successfully provisioned PVC "smithi187-lv-2-data-0wdspq" 2022-03-08T23:10:28.584 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:28.588265 I | op-osd: creating 1 new PVCs for device set "smithi187-lv-3" 2022-03-08T23:10:28.804 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:28.809357 I | op-osd: successfully provisioned PVC "smithi187-lv-3-data-0zsp69" 2022-03-08T23:10:28.805 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:28.809381 I | op-osd: creating 1 new PVCs for device set "smithi187-lv-4" 2022-03-08T23:10:28.982 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:28.987239 I | op-osd: successfully provisioned PVC "smithi187-lv-4-data-069kzv" 2022-03-08T23:10:29.003 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:29.008189 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:10:29.027 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:29.032854 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:10:29.401 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:29.405989 I | op-osd: started OSD provisioning job for PVC "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:10:29.401 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:29.406022 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:10:29.807 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:29.812514 I | op-osd: started OSD provisioning job for PVC "smithi107-lv-2-data-09kfs9" 2022-03-08T23:10:29.808 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:29.812558 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:10:30.219 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:30.223162 I | op-osd: started OSD provisioning job for PVC "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:10:30.220 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:30.223198 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:10:30.959 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:30.964110 I | op-osd: started OSD provisioning job for PVC "smithi107-lv-4-data-0mptlv" 2022-03-08T23:10:30.959 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:30.964144 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:10:31.345 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:31.350236 I | op-osd: started OSD provisioning job for PVC "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:10:31.345 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:31.350271 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0v2l2j" 2022-03-08T23:10:32.228 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:32.233698 I | op-osd: started OSD provisioning job for PVC "smithi155-lv-2-data-0v2l2j" 2022-03-08T23:10:32.229 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:32.233735 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:10:32.963 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:32.968414 I | op-osd: started OSD provisioning job for PVC "smithi155-lv-3-data-08gx6j" 2022-03-08T23:10:32.964 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:32.968439 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-4-data-04c4td" 2022-03-08T23:10:33.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:33.588320 I | op-osd: started OSD provisioning job for PVC "smithi155-lv-4-data-04c4td" 2022-03-08T23:10:33.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:33.588351 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:10:33.970 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:33.975627 I | op-osd: started OSD provisioning job for PVC "smithi187-lv-1-data-067hpz" 2022-03-08T23:10:33.971 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:33.975675 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:10:34.186 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:10:34.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:34.360631 I | op-osd: started OSD provisioning job for PVC "smithi187-lv-2-data-0wdspq" 2022-03-08T23:10:34.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:34.360665 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:10:34.699 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:34.704598 I | op-osd: started OSD provisioning job for PVC "smithi187-lv-3-data-0zsp69" 2022-03-08T23:10:34.700 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:34.704626 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:10:35.031 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.036351 I | op-osd: started OSD provisioning job for PVC "smithi187-lv-4-data-069kzv" 2022-03-08T23:10:35.031 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.036387 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:10:35.032 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.036395 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:10:35.373 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377773 I | op-osd: OSD orchestration status for PVC smithi107-lv-1-data-0mjtjg is "starting" 2022-03-08T23:10:35.373 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377820 I | op-osd: OSD orchestration status for PVC smithi107-lv-2-data-09kfs9 is "starting" 2022-03-08T23:10:35.373 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377833 I | op-osd: OSD orchestration status for PVC smithi107-lv-3-data-0r8x5p is "starting" 2022-03-08T23:10:35.374 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377844 I | op-osd: OSD orchestration status for PVC smithi107-lv-4-data-0mptlv is "starting" 2022-03-08T23:10:35.377 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377856 I | op-osd: OSD orchestration status for PVC smithi155-lv-1-data-0c6ssv is "starting" 2022-03-08T23:10:35.377 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377869 I | op-osd: OSD orchestration status for PVC smithi155-lv-2-data-0v2l2j is "starting" 2022-03-08T23:10:35.377 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377878 I | op-osd: OSD orchestration status for PVC smithi155-lv-3-data-08gx6j is "starting" 2022-03-08T23:10:35.377 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377887 I | op-osd: OSD orchestration status for PVC smithi155-lv-4-data-04c4td is "starting" 2022-03-08T23:10:35.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377895 I | op-osd: OSD orchestration status for PVC smithi187-lv-1-data-067hpz is "starting" 2022-03-08T23:10:35.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377903 I | op-osd: OSD orchestration status for PVC smithi187-lv-2-data-0wdspq is "starting" 2022-03-08T23:10:35.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377912 I | op-osd: OSD orchestration status for PVC smithi187-lv-3-data-0zsp69 is "starting" 2022-03-08T23:10:35.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:35.377921 I | op-osd: OSD orchestration status for PVC smithi187-lv-4-data-069kzv is "starting" 2022-03-08T23:10:35.623 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:10:35.643 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:10:35.644 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:10:45.645 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:10:46.923 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:10:46.943 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T23:10:46.944 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:10:47.552 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:47.553900 I | op-osd: OSD orchestration status for node smithi155-lv-2-data-0v2l2j is "orchestrating" 2022-03-08T23:10:49.275 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:49.278073 I | op-osd: OSD orchestration status for PVC smithi155-lv-2-data-0v2l2j is "orchestrating" 2022-03-08T23:10:51.471 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:51.473600 I | op-osd: OSD orchestration status for node smithi187-lv-2-data-0wdspq is "orchestrating" 2022-03-08T23:10:52.830 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:52.832469 I | op-osd: OSD orchestration status for PVC smithi187-lv-2-data-0wdspq is "orchestrating" 2022-03-08T23:10:54.029 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.031453 I | op-osd: OSD orchestration status for node smithi187-lv-4-data-069kzv is "orchestrating" 2022-03-08T23:10:54.180 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.183157 I | op-osd: OSD orchestration status for node smithi107-lv-2-data-09kfs9 is "orchestrating" 2022-03-08T23:10:54.405 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.407434 I | op-osd: OSD orchestration status for node smithi187-lv-3-data-0zsp69 is "orchestrating" 2022-03-08T23:10:54.530 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.532902 I | op-osd: OSD orchestration status for node smithi107-lv-4-data-0mptlv is "orchestrating" 2022-03-08T23:10:54.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.654868 I | op-osd: OSD orchestration status for PVC smithi107-lv-2-data-09kfs9 is "orchestrating" 2022-03-08T23:10:54.775 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.777928 I | op-osd: OSD orchestration status for PVC smithi187-lv-4-data-069kzv is "orchestrating" 2022-03-08T23:10:54.822 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.824642 I | op-osd: OSD orchestration status for PVC smithi187-lv-3-data-0zsp69 is "orchestrating" 2022-03-08T23:10:54.844 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.846658 I | op-osd: OSD orchestration status for node smithi155-lv-3-data-08gx6j is "orchestrating" 2022-03-08T23:10:54.852 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.855286 I | op-osd: OSD orchestration status for node smithi155-lv-4-data-04c4td is "orchestrating" 2022-03-08T23:10:54.927 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:54.930192 I | op-osd: OSD orchestration status for PVC smithi107-lv-4-data-0mptlv is "orchestrating" 2022-03-08T23:10:55.841 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:55.844221 I | op-osd: OSD orchestration status for PVC smithi155-lv-3-data-08gx6j is "orchestrating" 2022-03-08T23:10:55.906 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:55.906449 I | op-osd: OSD orchestration status for PVC smithi155-lv-4-data-04c4td is "orchestrating" 2022-03-08T23:10:56.945 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:10:57.083 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:57.083616 I | op-osd: OSD orchestration status for node smithi155-lv-1-data-0c6ssv is "orchestrating" 2022-03-08T23:10:57.278 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:57.280940 I | op-osd: OSD orchestration status for node smithi187-lv-1-data-067hpz is "orchestrating" 2022-03-08T23:10:57.396 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:57.398669 I | op-osd: OSD orchestration status for node smithi107-lv-3-data-0r8x5p is "orchestrating" 2022-03-08T23:10:57.472 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:57.474410 I | op-osd: OSD orchestration status for PVC smithi155-lv-1-data-0c6ssv is "orchestrating" 2022-03-08T23:10:58.362 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:58.364291 I | op-osd: OSD orchestration status for PVC smithi187-lv-1-data-067hpz is "orchestrating" 2022-03-08T23:10:58.363 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:58.364422 I | op-osd: OSD orchestration status for PVC smithi155-lv-2-data-0v2l2j is "completed" 2022-03-08T23:10:58.363 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:58.364480 I | op-osd: creating OSD 0 on PVC "smithi155-lv-2-data-0v2l2j" 2022-03-08T23:10:58.363 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:58.364534 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0v2l2j" 2022-03-08T23:10:58.477 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:10:58.504 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646781056,"num_remapped_pgs":0} 2022-03-08T23:10:58.505 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:10:58.690 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:58.692382 I | op-osd: OSD orchestration status for PVC smithi107-lv-3-data-0r8x5p is "orchestrating" 2022-03-08T23:10:58.696 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:58.695399 I | op-osd: OSD orchestration status for PVC smithi155-lv-4-data-04c4td is "failed" 2022-03-08T23:10:58.696 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:58.695452 E | op-osd: failed to provision OSD(s) on PVC smithi155-lv-4-data-04c4td. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-08T23:10:58.697 INFO:tasks.rook.operator.smithi107.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new afbaf122-e869-4755-bd2c-132ef168bd06 2022-03-08T23:10:58.697 INFO:tasks.rook.operator.smithi107.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:10:58.697 INFO:tasks.rook.operator.smithi107.stdout: stderr: 2022-03-08T23:10:57.984+0000 7fdc817b8700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:10:58.697 INFO:tasks.rook.operator.smithi107.stdout: stderr: 2022-03-08T23:10:57.984+0000 7fdc817b8700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:10:58.698 INFO:tasks.rook.operator.smithi107.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T23:10:58.698 INFO:tasks.rook.operator.smithi107.stdout:Traceback (most recent call last): 2022-03-08T23:10:58.698 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-08T23:10:58.698 INFO:tasks.rook.operator.smithi107.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-08T23:10:58.699 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-08T23:10:58.699 INFO:tasks.rook.operator.smithi107.stdout: self.main(self.argv) 2022-03-08T23:10:58.699 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-08T23:10:58.699 INFO:tasks.rook.operator.smithi107.stdout: return f(*a, **kw) 2022-03-08T23:10:58.699 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-08T23:10:58.700 INFO:tasks.rook.operator.smithi107.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-08T23:10:58.700 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:10:58.700 INFO:tasks.rook.operator.smithi107.stdout: instance.main() 2022-03-08T23:10:58.700 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-08T23:10:58.700 INFO:tasks.rook.operator.smithi107.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-08T23:10:58.701 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:10:58.701 INFO:tasks.rook.operator.smithi107.stdout: instance.main() 2022-03-08T23:10:58.701 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-08T23:10:58.701 INFO:tasks.rook.operator.smithi107.stdout: self.safe_prepare(self.args) 2022-03-08T23:10:58.701 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:10:58.702 INFO:tasks.rook.operator.smithi107.stdout: self.prepare() 2022-03-08T23:10:58.702 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:10:58.702 INFO:tasks.rook.operator.smithi107.stdout: return func(*a, **kw) 2022-03-08T23:10:58.702 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:10:58.702 INFO:tasks.rook.operator.smithi107.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:10:58.703 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:10:58.703 INFO:tasks.rook.operator.smithi107.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:10:58.703 INFO:tasks.rook.operator.smithi107.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-08T23:10:58.703 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,536][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi155-lv-4-data-04c4td raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_4 2022-03-08T23:10:58.704 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,537][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-08T23:10:58.704 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,543][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-08T23:10:58.704 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,543][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-08T23:10:58.704 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-08T23:10:58.704 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-08T23:10:58.705 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-08T23:10:58.705 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-08T23:10:58.705 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-08T23:10:58.705 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-08T23:10:58.705 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-08T23:10:58.706 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-08T23:10:58.706 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-08T23:10:58.706 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-08T23:10:58.706 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-08T23:10:58.706 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-08T23:10:58.707 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-08T23:10:58.707 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-08T23:10:58.707 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,554][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-08T23:10:58.707 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,555][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_4 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-08T23:10:58.707 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,835][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_4 2022-03-08T23:10:58.708 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,840][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_4" KNAME="dm-3" MAJ:MIN="253:3" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-08T23:10:58.708 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,841][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_4 2022-03-08T23:10:58.708 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,844][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_4 to check for BlueStore label 2022-03-08T23:10:58.708 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,844][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_4 to check for BlueStore label 2022-03-08T23:10:58.710 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,844][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_4 2022-03-08T23:10:58.710 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/vg_nvme/lv_4 /dev/mapper/vg_nvme-lv_4 /dev/disk/by-id/dm-uuid-LVM-cWY9FcOTf5vGwmjCNGsxy13SedDVHkb40HsYchqWcbT8ViLAuVMedyyEL8guIsLk /dev/disk/by-id/dm-name-vg_nvme-lv_4 2022-03-08T23:10:58.711 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-3 2022-03-08T23:10:58.711 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-3 2022-03-08T23:10:58.711 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-08T23:10:58.711 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_4 2022-03-08T23:10:58.711 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_4 2022-03-08T23:10:58.712 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-08T23:10:58.712 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-08T23:10:58.712 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-08T23:10:58.712 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-08T23:10:58.713 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-08T23:10:58.713 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UUID=LVM-cWY9FcOTf5vGwmjCNGsxy13SedDVHkb40HsYchqWcbT8ViLAuVMedyyEL8guIsLk 2022-03-08T23:10:58.713 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-08T23:10:58.713 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-08T23:10:58.714 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout MINOR=3 2022-03-08T23:10:58.714 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-08T23:10:58.714 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-08T23:10:58.714 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=194684842 2022-03-08T23:10:58.714 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,852][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-08T23:10:58.715 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,867][ceph_volume.process][INFO ] stdout AQCB4idi4FecMxAApRqKGzxjOp4gbDt65Kmn+w== 2022-03-08T23:10:58.715 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,867][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new afbaf122-e869-4755-bd2c-132ef168bd06 2022-03-08T23:10:58.715 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:10:58.715 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.process][INFO ] stderr 2022-03-08T23:10:57.984+0000 7fdc817b8700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:10:58.715 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.process][INFO ] stderr 2022-03-08T23:10:57.984+0000 7fdc817b8700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:10:58.716 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T23:10:58.716 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-08T23:10:58.716 INFO:tasks.rook.operator.smithi107.stdout:Traceback (most recent call last): 2022-03-08T23:10:58.716 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:10:58.717 INFO:tasks.rook.operator.smithi107.stdout: self.prepare() 2022-03-08T23:10:58.717 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:10:58.717 INFO:tasks.rook.operator.smithi107.stdout: return func(*a, **kw) 2022-03-08T23:10:58.717 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:10:58.717 INFO:tasks.rook.operator.smithi107.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:10:58.718 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:10:58.718 INFO:tasks.rook.operator.smithi107.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:10:58.718 INFO:tasks.rook.operator.smithi107.stdout:RuntimeError: Unable to create a new OSD id 2022-03-08T23:10:58.718 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,989][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-08T23:10:58.718 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,989][ceph_volume][ERROR ] exception caught by decorator 2022-03-08T23:10:58.719 INFO:tasks.rook.operator.smithi107.stdout:Traceback (most recent call last): 2022-03-08T23:10:58.719 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-08T23:10:58.719 INFO:tasks.rook.operator.smithi107.stdout: return f(*a, **kw) 2022-03-08T23:10:58.719 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-08T23:10:58.720 INFO:tasks.rook.operator.smithi107.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-08T23:10:58.720 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:10:58.720 INFO:tasks.rook.operator.smithi107.stdout: instance.main() 2022-03-08T23:10:58.720 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-08T23:10:58.720 INFO:tasks.rook.operator.smithi107.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-08T23:10:58.721 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:10:58.721 INFO:tasks.rook.operator.smithi107.stdout: instance.main() 2022-03-08T23:10:58.721 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-08T23:10:58.721 INFO:tasks.rook.operator.smithi107.stdout: self.safe_prepare(self.args) 2022-03-08T23:10:58.721 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:10:58.722 INFO:tasks.rook.operator.smithi107.stdout: self.prepare() 2022-03-08T23:10:58.722 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:10:58.722 INFO:tasks.rook.operator.smithi107.stdout: return func(*a, **kw) 2022-03-08T23:10:58.722 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:10:58.723 INFO:tasks.rook.operator.smithi107.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:10:58.723 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:10:58.723 INFO:tasks.rook.operator.smithi107.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:10:58.723 INFO:tasks.rook.operator.smithi107.stdout:RuntimeError: Unable to create a new OSD id 2022-03-08T23:10:58.723 INFO:tasks.rook.operator.smithi107.stdout:: exit status 1} 2022-03-08T23:10:58.868 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:58.870737 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi155": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:10:59.432 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:59.434029 I | op-osd: OSD orchestration status for PVC smithi187-lv-2-data-0wdspq is "completed" 2022-03-08T23:10:59.432 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:59.434065 I | op-osd: creating OSD 1 on PVC "smithi187-lv-2-data-0wdspq" 2022-03-08T23:10:59.432 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:10:59.434087 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:11:01.172 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:01.174722 I | op-osd: OSD orchestration status for node smithi107-lv-1-data-0mjtjg is "orchestrating" 2022-03-08T23:11:01.584 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:01.585980 I | op-osd: OSD orchestration status for PVC smithi107-lv-1-data-0mjtjg is "orchestrating" 2022-03-08T23:11:02.234 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:02.235601 I | op-osd: OSD orchestration status for PVC smithi187-lv-4-data-069kzv is "completed" 2022-03-08T23:11:02.234 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:02.235635 I | op-osd: creating OSD 2 on PVC "smithi187-lv-4-data-069kzv" 2022-03-08T23:11:02.234 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:02.235650 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:11:02.467 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:02.468006 I | op-osd: OSD orchestration status for node smithi155-lv-4-data-04c4td is "orchestrating" 2022-03-08T23:11:02.496 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:02.497432 I | op-osd: OSD orchestration status for PVC smithi187-lv-3-data-0zsp69 is "completed" 2022-03-08T23:11:02.500 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:02.497456 I | op-osd: creating OSD 3 on PVC "smithi187-lv-3-data-0zsp69" 2022-03-08T23:11:02.500 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:02.497471 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:11:02.853 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:02.852940 I | op-osd: OSD orchestration status for PVC smithi155-lv-4-data-04c4td is "orchestrating" 2022-03-08T23:11:04.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:04.334360 I | op-osd: OSD orchestration status for PVC smithi107-lv-2-data-09kfs9 is "completed" 2022-03-08T23:11:04.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:04.334402 I | op-osd: creating OSD 4 on PVC "smithi107-lv-2-data-09kfs9" 2022-03-08T23:11:04.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:04.334421 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:11:05.507 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:05.507112 I | op-osd: OSD orchestration status for PVC smithi107-lv-4-data-0mptlv is "completed" 2022-03-08T23:11:05.516 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:05.510321 I | op-osd: creating OSD 6 on PVC "smithi107-lv-4-data-0mptlv" 2022-03-08T23:11:05.518 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:05.510564 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:11:07.246 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:07.245147 I | op-osd: OSD orchestration status for PVC smithi155-lv-3-data-08gx6j is "completed" 2022-03-08T23:11:07.247 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:07.245176 I | op-osd: creating OSD 5 on PVC "smithi155-lv-3-data-08gx6j" 2022-03-08T23:11:07.248 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:07.245189 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:11:08.483 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:08.485234 I | op-osd: OSD orchestration status for PVC smithi187-lv-1-data-067hpz is "completed" 2022-03-08T23:11:08.505 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:11:08.565 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:08.486140 I | op-osd: creating OSD 8 on PVC "smithi187-lv-1-data-067hpz" 2022-03-08T23:11:08.566 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:08.486206 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:11:10.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:10.651192 I | op-osd: OSD orchestration status for PVC smithi155-lv-1-data-0c6ssv is "completed" 2022-03-08T23:11:10.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:10.651224 I | op-osd: creating OSD 7 on PVC "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:11:10.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:10.651238 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:11:10.681 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:11:10.699 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":15,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646781065,"num_remapped_pgs":0} 2022-03-08T23:11:10.699 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:11:11.911 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:11.911504 I | op-osd: OSD orchestration status for PVC smithi107-lv-3-data-0r8x5p is "completed" 2022-03-08T23:11:11.911 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:11.911532 I | op-osd: creating OSD 9 on PVC "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:11:11.912 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:11.911543 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:11:12.705 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:12.706547 I | op-osd: OSD orchestration status for PVC smithi155-lv-4-data-04c4td is "completed" 2022-03-08T23:11:12.709 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:12.708884 I | op-osd: not creating deployments for OSD prepare results found in ConfigMap "rook-ceph-osd-smithi155-lv-4-data-04c4td-status" which was already processed 2022-03-08T23:11:13.200 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:13.199781 I | op-osd: OSD orchestration status for PVC smithi107-lv-1-data-0mjtjg is "completed" 2022-03-08T23:11:13.201 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:13.199821 I | op-osd: creating OSD 10 on PVC "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:11:13.201 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:13.199833 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:11:14.317 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:14.317787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-08T23:11:14.317 INFO:tasks.rook.operator.smithi107.stdout:failed to provision OSD(s) on PVC smithi155-lv-4-data-04c4td. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-08T23:11:14.317 INFO:tasks.rook.operator.smithi107.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new afbaf122-e869-4755-bd2c-132ef168bd06 2022-03-08T23:11:14.318 INFO:tasks.rook.operator.smithi107.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:11:14.318 INFO:tasks.rook.operator.smithi107.stdout: stderr: 2022-03-08T23:10:57.984+0000 7fdc817b8700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:11:14.318 INFO:tasks.rook.operator.smithi107.stdout: stderr: 2022-03-08T23:10:57.984+0000 7fdc817b8700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:11:14.318 INFO:tasks.rook.operator.smithi107.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T23:11:14.319 INFO:tasks.rook.operator.smithi107.stdout:Traceback (most recent call last): 2022-03-08T23:11:14.319 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-08T23:11:14.319 INFO:tasks.rook.operator.smithi107.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-08T23:11:14.319 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-08T23:11:14.319 INFO:tasks.rook.operator.smithi107.stdout: self.main(self.argv) 2022-03-08T23:11:14.320 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-08T23:11:14.320 INFO:tasks.rook.operator.smithi107.stdout: return f(*a, **kw) 2022-03-08T23:11:14.320 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-08T23:11:14.320 INFO:tasks.rook.operator.smithi107.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-08T23:11:14.321 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:11:14.321 INFO:tasks.rook.operator.smithi107.stdout: instance.main() 2022-03-08T23:11:14.321 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-08T23:11:14.327 INFO:tasks.rook.operator.smithi107.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-08T23:11:14.327 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:11:14.328 INFO:tasks.rook.operator.smithi107.stdout: instance.main() 2022-03-08T23:11:14.328 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-08T23:11:14.328 INFO:tasks.rook.operator.smithi107.stdout: self.safe_prepare(self.args) 2022-03-08T23:11:14.328 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:11:14.329 INFO:tasks.rook.operator.smithi107.stdout: self.prepare() 2022-03-08T23:11:14.329 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:11:14.329 INFO:tasks.rook.operator.smithi107.stdout: return func(*a, **kw) 2022-03-08T23:11:14.329 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:11:14.329 INFO:tasks.rook.operator.smithi107.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:11:14.330 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:11:14.330 INFO:tasks.rook.operator.smithi107.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:11:14.330 INFO:tasks.rook.operator.smithi107.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-08T23:11:14.330 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,536][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi155-lv-4-data-04c4td raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_4 2022-03-08T23:11:14.330 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,537][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-08T23:11:14.331 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,543][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-08T23:11:14.331 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,543][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-08T23:11:14.331 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-08T23:11:14.331 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-08T23:11:14.332 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-08T23:11:14.332 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-08T23:11:14.332 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-08T23:11:14.332 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-08T23:11:14.332 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-08T23:11:14.333 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-08T23:11:14.333 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-08T23:11:14.333 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-08T23:11:14.333 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-08T23:11:14.333 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-08T23:11:14.334 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-08T23:11:14.334 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,544][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-08T23:11:14.334 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,554][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-08T23:11:14.334 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,555][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_4 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-08T23:11:14.334 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,835][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_4 2022-03-08T23:11:14.335 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,840][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_4" KNAME="dm-3" MAJ:MIN="253:3" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-08T23:11:14.335 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,841][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_4 2022-03-08T23:11:14.335 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,844][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_4 to check for BlueStore label 2022-03-08T23:11:14.335 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,844][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_4 to check for BlueStore label 2022-03-08T23:11:14.335 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,844][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_4 2022-03-08T23:11:14.336 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/vg_nvme/lv_4 /dev/mapper/vg_nvme-lv_4 /dev/disk/by-id/dm-uuid-LVM-cWY9FcOTf5vGwmjCNGsxy13SedDVHkb40HsYchqWcbT8ViLAuVMedyyEL8guIsLk /dev/disk/by-id/dm-name-vg_nvme-lv_4 2022-03-08T23:11:14.336 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-3 2022-03-08T23:11:14.336 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-3 2022-03-08T23:11:14.336 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-08T23:11:14.337 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_4 2022-03-08T23:11:14.337 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_4 2022-03-08T23:11:14.337 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-08T23:11:14.337 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-08T23:11:14.337 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-08T23:11:14.338 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-08T23:11:14.338 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-08T23:11:14.338 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_UUID=LVM-cWY9FcOTf5vGwmjCNGsxy13SedDVHkb40HsYchqWcbT8ViLAuVMedyyEL8guIsLk 2022-03-08T23:11:14.338 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-08T23:11:14.339 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-08T23:11:14.339 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout MINOR=3 2022-03-08T23:11:14.339 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-08T23:11:14.339 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-08T23:11:14.339 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,851][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=194684842 2022-03-08T23:11:14.340 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,852][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-08T23:11:14.340 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,867][ceph_volume.process][INFO ] stdout AQCB4idi4FecMxAApRqKGzxjOp4gbDt65Kmn+w== 2022-03-08T23:11:14.340 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,867][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new afbaf122-e869-4755-bd2c-132ef168bd06 2022-03-08T23:11:14.340 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-08T23:11:14.341 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.process][INFO ] stderr 2022-03-08T23:10:57.984+0000 7fdc817b8700 -1 failed for service _ceph-mon._tcp 2022-03-08T23:11:14.341 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.process][INFO ] stderr 2022-03-08T23:10:57.984+0000 7fdc817b8700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-08T23:11:14.341 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-08T23:11:14.341 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,988][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-08T23:11:14.341 INFO:tasks.rook.operator.smithi107.stdout:Traceback (most recent call last): 2022-03-08T23:11:14.342 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:11:14.342 INFO:tasks.rook.operator.smithi107.stdout: self.prepare() 2022-03-08T23:11:14.342 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:11:14.342 INFO:tasks.rook.operator.smithi107.stdout: return func(*a, **kw) 2022-03-08T23:11:14.343 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:11:14.343 INFO:tasks.rook.operator.smithi107.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:11:14.343 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:11:14.343 INFO:tasks.rook.operator.smithi107.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:11:14.343 INFO:tasks.rook.operator.smithi107.stdout:RuntimeError: Unable to create a new OSD id 2022-03-08T23:11:14.344 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,989][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-08T23:11:14.344 INFO:tasks.rook.operator.smithi107.stdout:[2022-03-08 23:10:57,989][ceph_volume][ERROR ] exception caught by decorator 2022-03-08T23:11:14.344 INFO:tasks.rook.operator.smithi107.stdout:Traceback (most recent call last): 2022-03-08T23:11:14.344 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-08T23:11:14.345 INFO:tasks.rook.operator.smithi107.stdout: return f(*a, **kw) 2022-03-08T23:11:14.345 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-08T23:11:14.345 INFO:tasks.rook.operator.smithi107.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-08T23:11:14.345 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:11:14.345 INFO:tasks.rook.operator.smithi107.stdout: instance.main() 2022-03-08T23:11:14.346 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-08T23:11:14.346 INFO:tasks.rook.operator.smithi107.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-08T23:11:14.346 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-08T23:11:14.346 INFO:tasks.rook.operator.smithi107.stdout: instance.main() 2022-03-08T23:11:14.347 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-08T23:11:14.347 INFO:tasks.rook.operator.smithi107.stdout: self.safe_prepare(self.args) 2022-03-08T23:11:14.347 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-08T23:11:14.347 INFO:tasks.rook.operator.smithi107.stdout: self.prepare() 2022-03-08T23:11:14.347 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-08T23:11:14.348 INFO:tasks.rook.operator.smithi107.stdout: return func(*a, **kw) 2022-03-08T23:11:14.348 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-08T23:11:14.348 INFO:tasks.rook.operator.smithi107.stdout: osd_fsid, json.dumps(secrets)) 2022-03-08T23:11:14.348 INFO:tasks.rook.operator.smithi107.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-08T23:11:14.349 INFO:tasks.rook.operator.smithi107.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-08T23:11:14.349 INFO:tasks.rook.operator.smithi107.stdout:RuntimeError: Unable to create a new OSD id 2022-03-08T23:11:14.349 INFO:tasks.rook.operator.smithi107.stdout:: exit status 1} 2022-03-08T23:11:14.349 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:14.324452 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:11:14.846 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:14.846721 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:11:15.801 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:15.801854 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:11:20.700 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:11:23.188 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:11:23.211 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":15,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646781065,"num_remapped_pgs":0} 2022-03-08T23:11:23.212 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:11:33.212 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:11:35.921 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:11:35.941 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":18,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646781065,"num_remapped_pgs":0} 2022-03-08T23:11:35.941 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T23:11:36.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:36.854837 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:11:36.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:36.854869 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:11:36.856 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:36.854880 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:11:36.919 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:36.915653 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:11:37.453 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:37.448685 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:11:37.487 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:37.487797 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:11:38.645 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:38.645320 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:11:40.358 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:40.358296 I | op-mon: start running mons 2022-03-08T23:11:40.513 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:40.513174 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:11:40.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:40.856141 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789","10.100.13.26:6789","10.97.212.176:6789"]}] data:a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:11:40.866 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:40.866345 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:11:40.869 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:40.866559 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:11:42.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:42.131950 I | op-mon: targeting the mon count 3 2022-03-08T23:11:42.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:42.378524 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:11:43.015 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:43.014656 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:11:43.015 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:43.014691 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:11:44.964 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:44.963590 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:11:44.964 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:44.963612 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:11:45.942 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:11:46.862 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:46.861463 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:11:46.862 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:46.861500 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:11:47.279 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:11:47.299 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":25,"num_osds":12,"num_up_osds":6,"osd_up_since":1646781106,"num_in_osds":12,"osd_in_since":1646781065,"num_remapped_pgs":0} 2022-03-08T23:11:47.300 INFO:tasks.rook: have 6/12 OSDs 2022-03-08T23:11:47.305 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:47.305652 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:11:47.306 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:47.305675 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:11:47.666 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:47.666353 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:11:47.667 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:47.666382 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:11:47.849 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:47.849290 I | op-mon: mon "a" endpoint is [v2:10.99.149.173:3300,v1:10.99.149.173:6789] 2022-03-08T23:11:49.434 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:49.433186 I | op-mon: mon "b" endpoint is [v2:10.100.13.26:3300,v1:10.100.13.26:6789] 2022-03-08T23:11:49.798 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:49.797574 I | op-mon: mon "c" endpoint is [v2:10.97.212.176:3300,v1:10.97.212.176:6789] 2022-03-08T23:11:49.824 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:49.824121 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789","10.100.13.26:6789","10.97.212.176:6789"]}] data:c=10.97.212.176:6789,a=10.99.149.173:6789,b=10.100.13.26:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:11:49.900 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:49.900263 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:11:49.901 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:49.900470 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:11:49.922 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:49.922183 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:11:49.929 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:49.929532 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:11:49.930 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:49.929555 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:11:50.448 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:50.448346 I | op-mon: mons running: [a b c] 2022-03-08T23:11:50.927 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:50.926525 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:11:50.931 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:50.931639 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:11:50.939 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:50.938935 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:11:50.939 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:50.938967 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:11:51.076 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:51.075914 I | op-mon: mons running: [a b c] 2022-03-08T23:11:51.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:51.492759 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:11:51.502 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:51.498886 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:11:51.508 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:51.508783 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:11:51.509 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:51.508807 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:11:51.649 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:51.648674 I | op-mon: mons running: [a b c] 2022-03-08T23:11:52.196 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:52.193553 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:11:52.196 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:52.193583 I | op-mon: mons created: 3 2022-03-08T23:11:52.597 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:52.596707 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T23:11:52.629 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:52.628551 I | op-mon: mons running: [a b c] 2022-03-08T23:11:53.063 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:53.062315 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:11:53.063 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:53.062349 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:11:53.486 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:53.485997 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:11:53.903 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:53.902769 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:11:54.413 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:54.413145 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:11:54.834 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:54.833400 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:11:54.834 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:54.833425 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:11:55.287 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:55.286648 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:11:55.816 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:55.815007 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:11:55.817 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:55.815047 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:11:55.817 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:55.815057 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:11:56.287 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:56.286388 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:11:56.373 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:56.372471 I | op-mgr: start running mgr 2022-03-08T23:11:56.373 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:56.372504 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:11:56.884 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:56.884132 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:11:56.896 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:56.896223 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:11:56.899 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:56.899289 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:11:57.058 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.057701 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:11:57.059 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.057818 I | op-mgr: successful modules: balancer 2022-03-08T23:11:57.075 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.074677 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:11:57.075 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.074707 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:11:57.121 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.120559 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:11:57.128 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125618 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-1" 2022-03-08T23:11:57.128 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125658 I | op-osd: OSD PVC "smithi107-lv-1-data-0mjtjg" already exists 2022-03-08T23:11:57.128 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125670 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-2" 2022-03-08T23:11:57.129 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125680 I | op-osd: OSD PVC "smithi107-lv-2-data-09kfs9" already exists 2022-03-08T23:11:57.129 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125691 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-3" 2022-03-08T23:11:57.129 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125701 I | op-osd: OSD PVC "smithi107-lv-3-data-0r8x5p" already exists 2022-03-08T23:11:57.129 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125711 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-4" 2022-03-08T23:11:57.129 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125721 I | op-osd: OSD PVC "smithi107-lv-4-data-0mptlv" already exists 2022-03-08T23:11:57.130 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125733 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-1" 2022-03-08T23:11:57.130 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125745 I | op-osd: OSD PVC "smithi155-lv-1-data-0c6ssv" already exists 2022-03-08T23:11:57.130 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125756 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-2" 2022-03-08T23:11:57.130 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125766 I | op-osd: OSD PVC "smithi155-lv-2-data-0v2l2j" already exists 2022-03-08T23:11:57.131 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125773 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-3" 2022-03-08T23:11:57.131 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125781 I | op-osd: OSD PVC "smithi155-lv-3-data-08gx6j" already exists 2022-03-08T23:11:57.131 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125787 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-4" 2022-03-08T23:11:57.131 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125797 I | op-osd: OSD PVC "smithi155-lv-4-data-04c4td" already exists 2022-03-08T23:11:57.131 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125804 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-1" 2022-03-08T23:11:57.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125813 I | op-osd: OSD PVC "smithi187-lv-1-data-067hpz" already exists 2022-03-08T23:11:57.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125825 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-2" 2022-03-08T23:11:57.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125835 I | op-osd: OSD PVC "smithi187-lv-2-data-0wdspq" already exists 2022-03-08T23:11:57.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125844 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-3" 2022-03-08T23:11:57.133 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125854 I | op-osd: OSD PVC "smithi187-lv-3-data-0zsp69" already exists 2022-03-08T23:11:57.133 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125863 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-4" 2022-03-08T23:11:57.133 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.125873 I | op-osd: OSD PVC "smithi187-lv-4-data-069kzv" already exists 2022-03-08T23:11:57.188 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187228 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:11:57.189 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187263 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-1-data-0mjtjg" because OSD daemon using the PVC already exists 2022-03-08T23:11:57.189 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187275 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:11:57.189 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187281 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-2-data-09kfs9" because OSD daemon using the PVC already exists 2022-03-08T23:11:57.189 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187294 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:11:57.189 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187302 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-3-data-0r8x5p" because OSD daemon using the PVC already exists 2022-03-08T23:11:57.190 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187308 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:11:57.190 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187311 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-4-data-0mptlv" because OSD daemon using the PVC already exists 2022-03-08T23:11:57.190 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187315 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:11:57.190 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187318 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-1-data-0c6ssv" because OSD daemon using the PVC already exists 2022-03-08T23:11:57.191 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187322 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0v2l2j" 2022-03-08T23:11:57.191 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187326 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-2-data-0v2l2j" because OSD daemon using the PVC already exists 2022-03-08T23:11:57.191 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187330 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:11:57.191 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187334 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-3-data-08gx6j" because OSD daemon using the PVC already exists 2022-03-08T23:11:57.192 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.187337 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-4-data-04c4td" 2022-03-08T23:11:57.302 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:11:57.458 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.452724 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-smithi155-lv-4-data-04c4td to start a new one 2022-03-08T23:11:57.503 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:57.478430 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi155-lv-4-data-04c4td still exists 2022-03-08T23:11:57.955 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:11:57.997 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":34,"num_osds":12,"num_up_osds":11,"osd_up_since":1646781116,"num_in_osds":12,"osd_in_since":1646781065,"num_remapped_pgs":1} 2022-03-08T23:11:57.997 INFO:tasks.rook: have 11/12 OSDs 2022-03-08T23:11:58.462 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:58.461386 I | op-mgr: successful modules: prometheus 2022-03-08T23:11:58.464 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:58.463440 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:11:58.465 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:11:58.463623 I | op-mgr: successful modules: dashboard 2022-03-08T23:12:00.412 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.411625 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:12:00.484 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.483671 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi155-lv-4-data-04c4td deleted 2022-03-08T23:12:00.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491198 I | op-osd: started OSD provisioning job for PVC "smithi155-lv-4-data-04c4td" 2022-03-08T23:12:00.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491232 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:12:00.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491242 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-1-data-067hpz" because OSD daemon using the PVC already exists 2022-03-08T23:12:00.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491251 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:12:00.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491261 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-2-data-0wdspq" because OSD daemon using the PVC already exists 2022-03-08T23:12:00.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491267 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:12:00.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491273 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-3-data-0zsp69" because OSD daemon using the PVC already exists 2022-03-08T23:12:00.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491280 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:12:00.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491286 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-4-data-069kzv" because OSD daemon using the PVC already exists 2022-03-08T23:12:00.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491292 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:12:00.495 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.491298 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:12:00.567 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.566575 I | op-osd: OSD orchestration status for PVC smithi155-lv-4-data-04c4td is "starting" 2022-03-08T23:12:00.669 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.668464 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:00.694 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.693523 I | op-osd: updating OSD 0 on PVC "smithi155-lv-2-data-0v2l2j" 2022-03-08T23:12:00.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.693559 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0v2l2j" 2022-03-08T23:12:00.872 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.871098 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:00.902 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.901553 I | op-osd: updating OSD 1 on PVC "smithi187-lv-2-data-0wdspq" 2022-03-08T23:12:00.903 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:00.901599 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:12:01.126 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:01.124980 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:01.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:01.131417 I | op-osd: updating OSD 10 on PVC "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:12:01.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:01.131439 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:12:01.304 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:01.303408 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:01.311 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:01.310202 I | op-osd: updating OSD 2 on PVC "smithi187-lv-4-data-069kzv" 2022-03-08T23:12:01.311 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:01.310233 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:12:02.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:02.004774 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:02.076 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:02.075868 I | op-osd: updating OSD 3 on PVC "smithi187-lv-3-data-0zsp69" 2022-03-08T23:12:02.077 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:02.075904 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:12:02.782 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:02.781416 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:02.927 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:02.926535 I | op-osd: updating OSD 4 on PVC "smithi107-lv-2-data-09kfs9" 2022-03-08T23:12:02.928 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:02.926553 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:12:03.590 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:03.588990 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:03.677 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:03.675910 I | op-osd: updating OSD 5 on PVC "smithi155-lv-3-data-08gx6j" 2022-03-08T23:12:03.677 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:03.675939 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:12:04.379 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:04.378246 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:04.478 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:04.476617 I | op-osd: updating OSD 6 on PVC "smithi107-lv-4-data-0mptlv" 2022-03-08T23:12:04.478 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:04.476658 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:12:05.181 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:05.180529 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:05.285 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:05.284920 I | op-osd: updating OSD 7 on PVC "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:12:05.286 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:05.284946 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:12:05.981 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:05.979711 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:06.077 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:06.076074 I | op-osd: updating OSD 8 on PVC "smithi187-lv-1-data-067hpz" 2022-03-08T23:12:06.078 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:06.076109 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:12:06.782 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:06.781790 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:12:06.876 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:06.875159 I | op-osd: updating OSD 9 on PVC "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:12:06.876 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:06.875201 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:12:07.999 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:12:08.506 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:08.527 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":37,"num_osds":12,"num_up_osds":11,"osd_up_since":1646781116,"num_in_osds":12,"osd_in_since":1646781065,"num_remapped_pgs":0} 2022-03-08T23:12:08.527 INFO:tasks.rook: have 11/12 OSDs 2022-03-08T23:12:08.728 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:08.726978 I | op-osd: OSD orchestration status for node smithi155-lv-4-data-04c4td is "orchestrating" 2022-03-08T23:12:08.793 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:08.792699 I | op-osd: OSD orchestration status for PVC smithi155-lv-4-data-04c4td is "orchestrating" 2022-03-08T23:12:09.875 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:09.874197 I | op-osd: OSD orchestration status for PVC smithi155-lv-4-data-04c4td is "completed" 2022-03-08T23:12:09.876 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:09.874227 I | op-osd: creating OSD 11 on PVC "smithi155-lv-4-data-04c4td" 2022-03-08T23:12:09.876 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:09.874243 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-4-data-04c4td" 2022-03-08T23:12:10.606 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:10.605029 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:12:10.606 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:10.605061 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:12:18.528 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:12:19.069 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:19.089 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":37,"num_osds":12,"num_up_osds":11,"osd_up_since":1646781116,"num_in_osds":12,"osd_in_since":1646781065,"num_remapped_pgs":0} 2022-03-08T23:12:19.090 INFO:tasks.rook: have 11/12 OSDs 2022-03-08T23:12:29.091 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:12:29.629 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:29.649 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":38,"num_osds":12,"num_up_osds":11,"osd_up_since":1646781116,"num_in_osds":12,"osd_in_since":1646781065,"num_remapped_pgs":0} 2022-03-08T23:12:29.650 INFO:tasks.rook: have 11/12 OSDs 2022-03-08T23:12:39.651 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd stat -f json 2022-03-08T23:12:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:40.149 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"num_osds":12,"num_up_osds":12,"osd_up_since":1646781151,"num_in_osds":12,"osd_in_since":1646781065,"num_remapped_pgs":0} 2022-03-08T23:12:40.150 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-08T23:12:40.151 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- cat /etc/ceph/ceph.conf 2022-03-08T23:12:40.315 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2022-03-08T23:12:40.316 INFO:teuthology.orchestra.run.smithi107.stdout:mon_host = 10.97.212.176:6789,10.99.149.173:6789,10.100.13.26:6789 2022-03-08T23:12:40.316 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:40.316 INFO:teuthology.orchestra.run.smithi107.stdout:[client.admin] 2022-03-08T23:12:40.316 INFO:teuthology.orchestra.run.smithi107.stdout:keyring = /etc/ceph/keyring 2022-03-08T23:12:40.320 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- cat /etc/ceph/keyring 2022-03-08T23:12:40.530 INFO:teuthology.orchestra.run.smithi107.stdout:[client.admin] 2022-03-08T23:12:40.531 INFO:teuthology.orchestra.run.smithi107.stdout:key = AQBu4Sdi3oP7ERAA3+u/T6pUiSVK5Sc67bXNWA== 2022-03-08T23:12:40.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2022-03-08T23:12:40.545 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph 2022-03-08T23:12:40.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph 2022-03-08T23:12:40.576 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:12:40.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:12:40.593 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:12:40.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:12:40.645 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:12:40.645 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:12:40.660 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:12:40.660 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:12:40.676 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:12:40.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T23:12:40.689 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:12:40.689 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/keyring 2022-03-08T23:12:40.700 INFO:tasks.rook:Setting up client nodes... 2022-03-08T23:12:40.700 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:12:41.396 INFO:teuthology.orchestra.run.smithi107.stdout:[client.a] 2022-03-08T23:12:41.397 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDp4idikfsjERAA0lrIlz0CDkYD7MT0MB8iVw== 2022-03-08T23:12:41.420 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2022-03-08T23:12:41.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-08T23:12:41.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-08T23:12:41.441 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:12:42.233 INFO:teuthology.orchestra.run.smithi107.stdout:[client.b] 2022-03-08T23:12:42.234 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDq4idi6KYEBxAAF1CBLhyygOgHhzFIgFLOuw== 2022-03-08T23:12:42.255 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-03-08T23:12:42.255 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-08T23:12:42.256 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-08T23:12:42.281 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T23:12:43.013 INFO:teuthology.orchestra.run.smithi107.stdout:[client.c] 2022-03-08T23:12:43.014 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDq4idiYFkxNhAAIHCir8i5cB+nAOA/MIutYw== 2022-03-08T23:12:43.036 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2022-03-08T23:12:43.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-08T23:12:43.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-08T23:12:43.058 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd dump --format=json 2022-03-08T23:12:43.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:43.540 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"fsid":"ed68b1de-23c1-4772-97c5-84bcc4ab6c90","created":"2022-03-08T23:06:36.211279+0000","modified":"2022-03-08T23:12:32.377904+0000","last_up_change":"2022-03-08T23:12:31.114328+0000","last_in_change":"2022-03-08T23:11:05.574796+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:11:42.747340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f445026f-c144-46b3-b63e-1c72b3927581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6800","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6801","nonce":104353392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6802","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6803","nonce":104353392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6806","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6807","nonce":104353392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6804","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6805","nonce":104353392}]},"public_addr":"10.251.83.206:6801/104353392","cluster_addr":"10.251.83.206:6803/104353392","heartbeat_back_addr":"10.251.83.206:6807/104353392","heartbeat_front_addr":"10.251.83.206:6805/104353392","state":["exists","up"]},{"osd":1,"uuid":"3c21a75f-8537-4482-b6a1-6496c1de8ff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6800","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6801","nonce":596731985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6802","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6803","nonce":596731985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6806","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6807","nonce":596731985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6804","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6805","nonce":596731985}]},"public_addr":"10.251.85.12:6801/596731985","cluster_addr":"10.251.85.12:6803/596731985","heartbeat_back_addr":"10.251.85.12:6807/596731985","heartbeat_front_addr":"10.251.85.12:6805/596731985","state":["exists","up"]},{"osd":2,"uuid":"7bf3a67c-95b1-47b7-91b0-6a20bc797a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6800","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6801","nonce":3506335308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6802","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6803","nonce":3506335308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6806","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6807","nonce":3506335308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6804","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6805","nonce":3506335308}]},"public_addr":"10.251.85.13:6801/3506335308","cluster_addr":"10.251.85.13:6803/3506335308","heartbeat_back_addr":"10.251.85.13:6807/3506335308","heartbeat_front_addr":"10.251.85.13:6805/3506335308","state":["exists","up"]},{"osd":3,"uuid":"a7be54be-24d2-49f5-9ef0-9ca9b981bbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6800","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6801","nonce":753941452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6802","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6803","nonce":753941452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6806","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6807","nonce":753941452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6804","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6805","nonce":753941452}]},"public_addr":"10.251.85.14:6801/753941452","cluster_addr":"10.251.85.14:6803/753941452","heartbeat_back_addr":"10.251.85.14:6807/753941452","heartbeat_front_addr":"10.251.85.14:6805/753941452","state":["exists","up"]},{"osd":4,"uuid":"c9b5433e-4687-434f-baba-716379c13df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6800","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6801","nonce":2145525464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6802","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6803","nonce":2145525464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6806","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6807","nonce":2145525464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6804","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6805","nonce":2145525464}]},"public_addr":"10.251.87.205:6801/2145525464","cluster_addr":"10.251.87.205:6803/2145525464","heartbeat_back_addr":"10.251.87.205:6807/2145525464","heartbeat_front_addr":"10.251.87.205:6805/2145525464","state":["exists","up"]},{"osd":5,"uuid":"b78addb0-719b-4384-91fc-b2cae579e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6800","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6801","nonce":2272951339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6802","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6803","nonce":2272951339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6806","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6807","nonce":2272951339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6804","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6805","nonce":2272951339}]},"public_addr":"10.251.83.207:6801/2272951339","cluster_addr":"10.251.83.207:6803/2272951339","heartbeat_back_addr":"10.251.83.207:6807/2272951339","heartbeat_front_addr":"10.251.83.207:6805/2272951339","state":["exists","up"]},{"osd":6,"uuid":"f01306db-b47a-4208-a3f9-2437fd7d1fda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6800","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6801","nonce":3881643472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6802","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6803","nonce":3881643472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6806","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6807","nonce":3881643472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6804","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6805","nonce":3881643472}]},"public_addr":"10.251.87.206:6801/3881643472","cluster_addr":"10.251.87.206:6803/3881643472","heartbeat_back_addr":"10.251.87.206:6807/3881643472","heartbeat_front_addr":"10.251.87.206:6805/3881643472","state":["exists","up"]},{"osd":7,"uuid":"f23418b9-60ce-4480-9d00-3bd92333065d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6800","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6801","nonce":1001046162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6802","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6803","nonce":1001046162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6806","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6807","nonce":1001046162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6804","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6805","nonce":1001046162}]},"public_addr":"10.251.83.208:6801/1001046162","cluster_addr":"10.251.83.208:6803/1001046162","heartbeat_back_addr":"10.251.83.208:6807/1001046162","heartbeat_front_addr":"10.251.83.208:6805/1001046162","state":["exists","up"]},{"osd":8,"uuid":"da2a71cb-3de8-4a63-b9a1-5865b558fd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6800","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6801","nonce":750006176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6802","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6803","nonce":750006176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6806","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6807","nonce":750006176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6804","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6805","nonce":750006176}]},"public_addr":"10.251.85.15:6801/750006176","cluster_addr":"10.251.85.15:6803/750006176","heartbeat_back_addr":"10.251.85.15:6807/750006176","heartbeat_front_addr":"10.251.85.15:6805/750006176","state":["exists","up"]},{"osd":9,"uuid":"de56f0f8-5803-4391-bedb-f66bad4007d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6800","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6801","nonce":1524028151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6802","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6803","nonce":1524028151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6806","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6807","nonce":1524028151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6804","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6805","nonce":1524028151}]},"public_addr":"10.251.87.207:6801/1524028151","cluster_addr":"10.251.87.207:6803/1524028151","heartbeat_back_addr":"10.251.87.207:6807/1524028151","heartbeat_front_addr":"10.251.87.207:6805/1524028151","state":["exists","up"]},{"osd":10,"uuid":"19bb089f-121f-4b5a-9131-848e82480245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6800","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6801","nonce":2798536935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6802","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6803","nonce":2798536935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6806","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6807","nonce":2798536935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6804","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6805","nonce":2798536935}]},"public_addr":"10.251.87.208:6801/2798536935","cluster_addr":"10.251.87.208:6803/2798536935","heartbeat_back_addr":"10.251.87.208:6807/2798536935","heartbeat_front_addr":"10.251.87.208:6805/2798536935","state":["exists","up"]},{"osd":11,"uuid":"e56e032f-c5f8-4fa0-b8e8-52ad31074a45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6800","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6801","nonce":2390044617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6802","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6803","nonce":2390044617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6806","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6807","nonce":2390044617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6804","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6805","nonce":2390044617}]},"public_addr":"10.251.83.211:6801/2390044617","cluster_addr":"10.251.83.211:6803/2390044617","heartbeat_back_addr":"10.251.83.211:6807/2390044617","heartbeat_front_addr":"10.251.83.211:6805/2390044617","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:29.865954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:34.470969+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:36.791550+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:38.914363+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:45.488306+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:40.868285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:51.658773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:47.574238+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:42.601017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:54.560537+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:54.521278+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:12:29.474257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.251.83.197:0/3753670617":"2022-03-09T23:09:52.231909+0000","10.251.83.197:6801/1184013250":"2022-03-09T23:09:52.231909+0000","10.251.83.197:6800/1184013250":"2022-03-09T23:09:52.231909+0000","10.251.83.197:0/824117151":"2022-03-09T23:09:52.231909+0000","10.251.83.197:0/1624690280":"2022-03-09T23:09:52.231909+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:12:43.560 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-08T23:11:42.747340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-08T23:12:43.561 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd pool get .mgr pg_num 2022-03-08T23:12:44.090 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2022-03-08T23:12:44.113 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-08T23:12:44.114 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-08T23:12:44.115 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph mgr dump --format=json 2022-03-08T23:12:44.687 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:44.704 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"active_gid":24194,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.197:6800","nonce":2104597375},{"type":"v1","addr":"10.251.83.197:6801","nonce":2104597375}]},"active_addr":"10.251.83.197:6801/2104597375","active_change":"2022-03-08T23:09:58.846789+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.251.83.197:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.251.83.197:0","nonce":1997926701}]},{"addrvec":[{"type":"v2","addr":"10.251.83.197:0","nonce":1596552844}]},{"addrvec":[{"type":"v2","addr":"10.251.83.197:0","nonce":2137091996}]},{"addrvec":[{"type":"v2","addr":"10.251.83.197:0","nonce":605557277}]}]}} 2022-03-08T23:12:44.708 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-08T23:12:44.709 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-08T23:12:44.709 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd dump --format=json 2022-03-08T23:12:45.222 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:45.222 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"fsid":"ed68b1de-23c1-4772-97c5-84bcc4ab6c90","created":"2022-03-08T23:06:36.211279+0000","modified":"2022-03-08T23:12:32.377904+0000","last_up_change":"2022-03-08T23:12:31.114328+0000","last_in_change":"2022-03-08T23:11:05.574796+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:11:42.747340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f445026f-c144-46b3-b63e-1c72b3927581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6800","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6801","nonce":104353392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6802","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6803","nonce":104353392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6806","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6807","nonce":104353392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6804","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6805","nonce":104353392}]},"public_addr":"10.251.83.206:6801/104353392","cluster_addr":"10.251.83.206:6803/104353392","heartbeat_back_addr":"10.251.83.206:6807/104353392","heartbeat_front_addr":"10.251.83.206:6805/104353392","state":["exists","up"]},{"osd":1,"uuid":"3c21a75f-8537-4482-b6a1-6496c1de8ff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6800","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6801","nonce":596731985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6802","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6803","nonce":596731985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6806","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6807","nonce":596731985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6804","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6805","nonce":596731985}]},"public_addr":"10.251.85.12:6801/596731985","cluster_addr":"10.251.85.12:6803/596731985","heartbeat_back_addr":"10.251.85.12:6807/596731985","heartbeat_front_addr":"10.251.85.12:6805/596731985","state":["exists","up"]},{"osd":2,"uuid":"7bf3a67c-95b1-47b7-91b0-6a20bc797a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6800","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6801","nonce":3506335308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6802","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6803","nonce":3506335308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6806","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6807","nonce":3506335308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6804","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6805","nonce":3506335308}]},"public_addr":"10.251.85.13:6801/3506335308","cluster_addr":"10.251.85.13:6803/3506335308","heartbeat_back_addr":"10.251.85.13:6807/3506335308","heartbeat_front_addr":"10.251.85.13:6805/3506335308","state":["exists","up"]},{"osd":3,"uuid":"a7be54be-24d2-49f5-9ef0-9ca9b981bbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6800","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6801","nonce":753941452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6802","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6803","nonce":753941452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6806","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6807","nonce":753941452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6804","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6805","nonce":753941452}]},"public_addr":"10.251.85.14:6801/753941452","cluster_addr":"10.251.85.14:6803/753941452","heartbeat_back_addr":"10.251.85.14:6807/753941452","heartbeat_front_addr":"10.251.85.14:6805/753941452","state":["exists","up"]},{"osd":4,"uuid":"c9b5433e-4687-434f-baba-716379c13df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6800","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6801","nonce":2145525464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6802","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6803","nonce":2145525464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6806","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6807","nonce":2145525464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6804","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6805","nonce":2145525464}]},"public_addr":"10.251.87.205:6801/2145525464","cluster_addr":"10.251.87.205:6803/2145525464","heartbeat_back_addr":"10.251.87.205:6807/2145525464","heartbeat_front_addr":"10.251.87.205:6805/2145525464","state":["exists","up"]},{"osd":5,"uuid":"b78addb0-719b-4384-91fc-b2cae579e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6800","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6801","nonce":2272951339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6802","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6803","nonce":2272951339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6806","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6807","nonce":2272951339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6804","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6805","nonce":2272951339}]},"public_addr":"10.251.83.207:6801/2272951339","cluster_addr":"10.251.83.207:6803/2272951339","heartbeat_back_addr":"10.251.83.207:6807/2272951339","heartbeat_front_addr":"10.251.83.207:6805/2272951339","state":["exists","up"]},{"osd":6,"uuid":"f01306db-b47a-4208-a3f9-2437fd7d1fda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6800","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6801","nonce":3881643472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6802","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6803","nonce":3881643472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6806","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6807","nonce":3881643472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6804","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6805","nonce":3881643472}]},"public_addr":"10.251.87.206:6801/3881643472","cluster_addr":"10.251.87.206:6803/3881643472","heartbeat_back_addr":"10.251.87.206:6807/3881643472","heartbeat_front_addr":"10.251.87.206:6805/3881643472","state":["exists","up"]},{"osd":7,"uuid":"f23418b9-60ce-4480-9d00-3bd92333065d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6800","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6801","nonce":1001046162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6802","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6803","nonce":1001046162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6806","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6807","nonce":1001046162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6804","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6805","nonce":1001046162}]},"public_addr":"10.251.83.208:6801/1001046162","cluster_addr":"10.251.83.208:6803/1001046162","heartbeat_back_addr":"10.251.83.208:6807/1001046162","heartbeat_front_addr":"10.251.83.208:6805/1001046162","state":["exists","up"]},{"osd":8,"uuid":"da2a71cb-3de8-4a63-b9a1-5865b558fd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6800","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6801","nonce":750006176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6802","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6803","nonce":750006176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6806","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6807","nonce":750006176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6804","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6805","nonce":750006176}]},"public_addr":"10.251.85.15:6801/750006176","cluster_addr":"10.251.85.15:6803/750006176","heartbeat_back_addr":"10.251.85.15:6807/750006176","heartbeat_front_addr":"10.251.85.15:6805/750006176","state":["exists","up"]},{"osd":9,"uuid":"de56f0f8-5803-4391-bedb-f66bad4007d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6800","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6801","nonce":1524028151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6802","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6803","nonce":1524028151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6806","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6807","nonce":1524028151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6804","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6805","nonce":1524028151}]},"public_addr":"10.251.87.207:6801/1524028151","cluster_addr":"10.251.87.207:6803/1524028151","heartbeat_back_addr":"10.251.87.207:6807/1524028151","heartbeat_front_addr":"10.251.87.207:6805/1524028151","state":["exists","up"]},{"osd":10,"uuid":"19bb089f-121f-4b5a-9131-848e82480245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6800","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6801","nonce":2798536935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6802","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6803","nonce":2798536935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6806","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6807","nonce":2798536935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6804","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6805","nonce":2798536935}]},"public_addr":"10.251.87.208:6801/2798536935","cluster_addr":"10.251.87.208:6803/2798536935","heartbeat_back_addr":"10.251.87.208:6807/2798536935","heartbeat_front_addr":"10.251.87.208:6805/2798536935","state":["exists","up"]},{"osd":11,"uuid":"e56e032f-c5f8-4fa0-b8e8-52ad31074a45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6800","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6801","nonce":2390044617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6802","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6803","nonce":2390044617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6806","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6807","nonce":2390044617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6804","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6805","nonce":2390044617}]},"public_addr":"10.251.83.211:6801/2390044617","cluster_addr":"10.251.83.211:6803/2390044617","heartbeat_back_addr":"10.251.83.211:6807/2390044617","heartbeat_front_addr":"10.251.83.211:6805/2390044617","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:29.865954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:34.470969+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:36.791550+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:38.914363+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:45.488306+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:40.868285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:51.658773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:47.574238+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:42.601017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:54.560537+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:54.521278+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:12:29.474257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.251.83.197:0/3753670617":"2022-03-09T23:09:52.231909+0000","10.251.83.197:6801/1184013250":"2022-03-09T23:09:52.231909+0000","10.251.83.197:6800/1184013250":"2022-03-09T23:09:52.231909+0000","10.251.83.197:0/824117151":"2022-03-09T23:09:52.231909+0000","10.251.83.197:0/1624690280":"2022-03-09T23:09:52.231909+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:12:45.243 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-08T23:12:45.244 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd dump --format=json 2022-03-08T23:12:45.754 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:45.755 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"fsid":"ed68b1de-23c1-4772-97c5-84bcc4ab6c90","created":"2022-03-08T23:06:36.211279+0000","modified":"2022-03-08T23:12:32.377904+0000","last_up_change":"2022-03-08T23:12:31.114328+0000","last_in_change":"2022-03-08T23:11:05.574796+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T23:11:42.747340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f445026f-c144-46b3-b63e-1c72b3927581","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6800","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6801","nonce":104353392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6802","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6803","nonce":104353392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6806","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6807","nonce":104353392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.206:6804","nonce":104353392},{"type":"v1","addr":"10.251.83.206:6805","nonce":104353392}]},"public_addr":"10.251.83.206:6801/104353392","cluster_addr":"10.251.83.206:6803/104353392","heartbeat_back_addr":"10.251.83.206:6807/104353392","heartbeat_front_addr":"10.251.83.206:6805/104353392","state":["exists","up"]},{"osd":1,"uuid":"3c21a75f-8537-4482-b6a1-6496c1de8ff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6800","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6801","nonce":596731985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6802","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6803","nonce":596731985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6806","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6807","nonce":596731985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.12:6804","nonce":596731985},{"type":"v1","addr":"10.251.85.12:6805","nonce":596731985}]},"public_addr":"10.251.85.12:6801/596731985","cluster_addr":"10.251.85.12:6803/596731985","heartbeat_back_addr":"10.251.85.12:6807/596731985","heartbeat_front_addr":"10.251.85.12:6805/596731985","state":["exists","up"]},{"osd":2,"uuid":"7bf3a67c-95b1-47b7-91b0-6a20bc797a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6800","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6801","nonce":3506335308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6802","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6803","nonce":3506335308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6806","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6807","nonce":3506335308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.13:6804","nonce":3506335308},{"type":"v1","addr":"10.251.85.13:6805","nonce":3506335308}]},"public_addr":"10.251.85.13:6801/3506335308","cluster_addr":"10.251.85.13:6803/3506335308","heartbeat_back_addr":"10.251.85.13:6807/3506335308","heartbeat_front_addr":"10.251.85.13:6805/3506335308","state":["exists","up"]},{"osd":3,"uuid":"a7be54be-24d2-49f5-9ef0-9ca9b981bbd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6800","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6801","nonce":753941452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6802","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6803","nonce":753941452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6806","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6807","nonce":753941452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.14:6804","nonce":753941452},{"type":"v1","addr":"10.251.85.14:6805","nonce":753941452}]},"public_addr":"10.251.85.14:6801/753941452","cluster_addr":"10.251.85.14:6803/753941452","heartbeat_back_addr":"10.251.85.14:6807/753941452","heartbeat_front_addr":"10.251.85.14:6805/753941452","state":["exists","up"]},{"osd":4,"uuid":"c9b5433e-4687-434f-baba-716379c13df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6800","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6801","nonce":2145525464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6802","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6803","nonce":2145525464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6806","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6807","nonce":2145525464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.205:6804","nonce":2145525464},{"type":"v1","addr":"10.251.87.205:6805","nonce":2145525464}]},"public_addr":"10.251.87.205:6801/2145525464","cluster_addr":"10.251.87.205:6803/2145525464","heartbeat_back_addr":"10.251.87.205:6807/2145525464","heartbeat_front_addr":"10.251.87.205:6805/2145525464","state":["exists","up"]},{"osd":5,"uuid":"b78addb0-719b-4384-91fc-b2cae579e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6800","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6801","nonce":2272951339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6802","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6803","nonce":2272951339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6806","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6807","nonce":2272951339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.207:6804","nonce":2272951339},{"type":"v1","addr":"10.251.83.207:6805","nonce":2272951339}]},"public_addr":"10.251.83.207:6801/2272951339","cluster_addr":"10.251.83.207:6803/2272951339","heartbeat_back_addr":"10.251.83.207:6807/2272951339","heartbeat_front_addr":"10.251.83.207:6805/2272951339","state":["exists","up"]},{"osd":6,"uuid":"f01306db-b47a-4208-a3f9-2437fd7d1fda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6800","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6801","nonce":3881643472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6802","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6803","nonce":3881643472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6806","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6807","nonce":3881643472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.206:6804","nonce":3881643472},{"type":"v1","addr":"10.251.87.206:6805","nonce":3881643472}]},"public_addr":"10.251.87.206:6801/3881643472","cluster_addr":"10.251.87.206:6803/3881643472","heartbeat_back_addr":"10.251.87.206:6807/3881643472","heartbeat_front_addr":"10.251.87.206:6805/3881643472","state":["exists","up"]},{"osd":7,"uuid":"f23418b9-60ce-4480-9d00-3bd92333065d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6800","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6801","nonce":1001046162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6802","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6803","nonce":1001046162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6806","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6807","nonce":1001046162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.208:6804","nonce":1001046162},{"type":"v1","addr":"10.251.83.208:6805","nonce":1001046162}]},"public_addr":"10.251.83.208:6801/1001046162","cluster_addr":"10.251.83.208:6803/1001046162","heartbeat_back_addr":"10.251.83.208:6807/1001046162","heartbeat_front_addr":"10.251.83.208:6805/1001046162","state":["exists","up"]},{"osd":8,"uuid":"da2a71cb-3de8-4a63-b9a1-5865b558fd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6800","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6801","nonce":750006176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6802","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6803","nonce":750006176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6806","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6807","nonce":750006176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.85.15:6804","nonce":750006176},{"type":"v1","addr":"10.251.85.15:6805","nonce":750006176}]},"public_addr":"10.251.85.15:6801/750006176","cluster_addr":"10.251.85.15:6803/750006176","heartbeat_back_addr":"10.251.85.15:6807/750006176","heartbeat_front_addr":"10.251.85.15:6805/750006176","state":["exists","up"]},{"osd":9,"uuid":"de56f0f8-5803-4391-bedb-f66bad4007d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6800","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6801","nonce":1524028151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6802","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6803","nonce":1524028151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6806","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6807","nonce":1524028151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.207:6804","nonce":1524028151},{"type":"v1","addr":"10.251.87.207:6805","nonce":1524028151}]},"public_addr":"10.251.87.207:6801/1524028151","cluster_addr":"10.251.87.207:6803/1524028151","heartbeat_back_addr":"10.251.87.207:6807/1524028151","heartbeat_front_addr":"10.251.87.207:6805/1524028151","state":["exists","up"]},{"osd":10,"uuid":"19bb089f-121f-4b5a-9131-848e82480245","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6800","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6801","nonce":2798536935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6802","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6803","nonce":2798536935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6806","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6807","nonce":2798536935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.87.208:6804","nonce":2798536935},{"type":"v1","addr":"10.251.87.208:6805","nonce":2798536935}]},"public_addr":"10.251.87.208:6801/2798536935","cluster_addr":"10.251.87.208:6803/2798536935","heartbeat_back_addr":"10.251.87.208:6807/2798536935","heartbeat_front_addr":"10.251.87.208:6805/2798536935","state":["exists","up"]},{"osd":11,"uuid":"e56e032f-c5f8-4fa0-b8e8-52ad31074a45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6800","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6801","nonce":2390044617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6802","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6803","nonce":2390044617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6806","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6807","nonce":2390044617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.83.211:6804","nonce":2390044617},{"type":"v1","addr":"10.251.83.211:6805","nonce":2390044617}]},"public_addr":"10.251.83.211:6801/2390044617","cluster_addr":"10.251.83.211:6803/2390044617","heartbeat_back_addr":"10.251.83.211:6807/2390044617","heartbeat_front_addr":"10.251.83.211:6805/2390044617","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:29.865954+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:34.470969+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:36.791550+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:38.914363+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:45.488306+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:40.868285+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:51.658773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:47.574238+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:42.601017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:54.560537+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:11:54.521278+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T23:12:29.474257+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.251.83.197:0/3753670617":"2022-03-09T23:09:52.231909+0000","10.251.83.197:6801/1184013250":"2022-03-09T23:09:52.231909+0000","10.251.83.197:6800/1184013250":"2022-03-09T23:09:52.231909+0000","10.251.83.197:0/824117151":"2022-03-09T23:09:52.231909+0000","10.251.83.197:0/1624690280":"2022-03-09T23:09:52.231909+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T23:12:45.774 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.0 flush_pg_stats 2022-03-08T23:12:45.774 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.1 flush_pg_stats 2022-03-08T23:12:45.775 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.2 flush_pg_stats 2022-03-08T23:12:45.776 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.3 flush_pg_stats 2022-03-08T23:12:45.776 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.4 flush_pg_stats 2022-03-08T23:12:45.777 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.5 flush_pg_stats 2022-03-08T23:12:45.778 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.6 flush_pg_stats 2022-03-08T23:12:45.778 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.7 flush_pg_stats 2022-03-08T23:12:45.779 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.8 flush_pg_stats 2022-03-08T23:12:45.780 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.9 flush_pg_stats 2022-03-08T23:12:45.780 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.10 flush_pg_stats 2022-03-08T23:12:45.781 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph tell osd.11 flush_pg_stats 2022-03-08T23:12:46.317 INFO:teuthology.orchestra.run.smithi107.stdout:107374182416 2022-03-08T23:12:46.318 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.5 2022-03-08T23:12:46.628 INFO:teuthology.orchestra.run.smithi107.stdout:133143986189 2022-03-08T23:12:46.629 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.6 2022-03-08T23:12:46.632 INFO:teuthology.orchestra.run.smithi107.stdout:120259084303 2022-03-08T23:12:46.632 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.4 2022-03-08T23:12:46.641 INFO:teuthology.orchestra.run.smithi107.stdout:90194313232 2022-03-08T23:12:46.641 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.2 2022-03-08T23:12:46.642 INFO:teuthology.orchestra.run.smithi107.stdout:81604378642 2022-03-08T23:12:46.643 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.0 2022-03-08T23:12:46.649 INFO:teuthology.orchestra.run.smithi107.stdout:124554051597 2022-03-08T23:12:46.649 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.7 2022-03-08T23:12:46.657 INFO:teuthology.orchestra.run.smithi107.stdout:103079215119 2022-03-08T23:12:46.657 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.8 2022-03-08T23:12:46.664 INFO:teuthology.orchestra.run.smithi107.stdout:141733920780 2022-03-08T23:12:46.665 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.9 2022-03-08T23:12:46.688 INFO:teuthology.orchestra.run.smithi107.stdout:141733920779 2022-03-08T23:12:46.688 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.10 2022-03-08T23:12:46.743 INFO:teuthology.orchestra.run.smithi107.stdout:85899345937 2022-03-08T23:12:46.744 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.1 2022-03-08T23:12:46.914 INFO:teuthology.orchestra.run.smithi107.stdout:171798691845 2022-03-08T23:12:46.915 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.11 2022-03-08T23:12:47.024 INFO:teuthology.orchestra.run.smithi107.stdout:94489280528 2022-03-08T23:12:47.024 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.3 2022-03-08T23:12:47.262 INFO:teuthology.orchestra.run.smithi107.stdout:107374182415 2022-03-08T23:12:47.424 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182416 got 107374182415 for osd.5 2022-03-08T23:12:47.455 INFO:teuthology.orchestra.run.smithi107.stdout:120259084301 2022-03-08T23:12:47.619 INFO:teuthology.orchestra.run.smithi107.stdout:124554051596 2022-03-08T23:12:47.633 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084303 got 120259084301 for osd.4 2022-03-08T23:12:47.731 INFO:teuthology.orchestra.run.smithi107.stdout:141733920779 2022-03-08T23:12:47.740 INFO:teuthology.orchestra.run.smithi107.stdout:103079215117 2022-03-08T23:12:47.745 INFO:teuthology.orchestra.run.smithi107.stdout:141733920778 2022-03-08T23:12:47.748 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051597 got 124554051596 for osd.7 2022-03-08T23:12:47.769 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920780 got 141733920779 for osd.9 2022-03-08T23:12:47.780 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920779 got 141733920778 for osd.10 2022-03-08T23:12:47.790 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215119 got 103079215117 for osd.8 2022-03-08T23:12:47.903 INFO:teuthology.orchestra.run.smithi107.stdout:133143986188 2022-03-08T23:12:47.925 INFO:tasks.rook.ceph_manager.ceph:need seq 133143986189 got 133143986188 for osd.6 2022-03-08T23:12:47.937 INFO:teuthology.orchestra.run.smithi107.stdout:90194313231 2022-03-08T23:12:47.994 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313232 got 90194313231 for osd.2 2022-03-08T23:12:48.007 INFO:teuthology.orchestra.run.smithi107.stdout:81604378641 2022-03-08T23:12:48.034 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378642 got 81604378641 for osd.0 2022-03-08T23:12:48.088 INFO:teuthology.orchestra.run.smithi107.stdout:171798691844 2022-03-08T23:12:48.106 INFO:tasks.rook.ceph_manager.ceph:need seq 171798691845 got 171798691844 for osd.11 2022-03-08T23:12:48.124 INFO:teuthology.orchestra.run.smithi107.stdout:94489280527 2022-03-08T23:12:48.142 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280528 got 94489280527 for osd.3 2022-03-08T23:12:48.206 INFO:teuthology.orchestra.run.smithi107.stdout:85899345936 2022-03-08T23:12:48.226 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345937 got 85899345936 for osd.1 2022-03-08T23:12:48.426 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.5 2022-03-08T23:12:48.635 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.4 2022-03-08T23:12:48.749 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.7 2022-03-08T23:12:48.770 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.9 2022-03-08T23:12:48.781 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.10 2022-03-08T23:12:48.791 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.8 2022-03-08T23:12:48.926 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.6 2022-03-08T23:12:48.995 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.2 2022-03-08T23:12:49.028 INFO:teuthology.orchestra.run.smithi107.stdout:107374182416 2022-03-08T23:12:49.035 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.0 2022-03-08T23:12:49.094 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182416 got 107374182416 for osd.5 2022-03-08T23:12:49.094 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:49.107 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.11 2022-03-08T23:12:49.142 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.3 2022-03-08T23:12:49.227 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph osd last-stat-seq osd.1 2022-03-08T23:12:49.479 INFO:teuthology.orchestra.run.smithi107.stdout:120259084303 2022-03-08T23:12:49.533 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084303 got 120259084303 for osd.4 2022-03-08T23:12:49.534 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:49.717 INFO:teuthology.orchestra.run.smithi107.stdout:141733920781 2022-03-08T23:12:49.762 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920780 got 141733920781 for osd.9 2022-03-08T23:12:49.762 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:49.768 INFO:teuthology.orchestra.run.smithi107.stdout:124554051597 2022-03-08T23:12:49.782 INFO:teuthology.orchestra.run.smithi107.stdout:141733920780 2022-03-08T23:12:49.786 INFO:teuthology.orchestra.run.smithi107.stdout:103079215119 2022-03-08T23:12:49.805 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051597 got 124554051597 for osd.7 2022-03-08T23:12:49.806 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:49.816 INFO:tasks.rook.ceph_manager.ceph:need seq 141733920779 got 141733920780 for osd.10 2022-03-08T23:12:49.816 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:49.829 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215119 got 103079215119 for osd.8 2022-03-08T23:12:49.829 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:49.939 INFO:teuthology.orchestra.run.smithi107.stdout:81604378643 2022-03-08T23:12:49.956 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378642 got 81604378643 for osd.0 2022-03-08T23:12:49.956 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:50.000 INFO:teuthology.orchestra.run.smithi107.stdout:90194313233 2022-03-08T23:12:50.019 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313232 got 90194313233 for osd.2 2022-03-08T23:12:50.020 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:50.046 INFO:teuthology.orchestra.run.smithi107.stdout:133143986189 2022-03-08T23:12:50.068 INFO:tasks.rook.ceph_manager.ceph:need seq 133143986189 got 133143986189 for osd.6 2022-03-08T23:12:50.069 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:50.072 INFO:teuthology.orchestra.run.smithi107.stdout:85899345937 2022-03-08T23:12:50.089 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345937 got 85899345937 for osd.1 2022-03-08T23:12:50.090 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:50.121 INFO:teuthology.orchestra.run.smithi107.stdout:171798691846 2022-03-08T23:12:50.138 INFO:tasks.rook.ceph_manager.ceph:need seq 171798691845 got 171798691846 for osd.11 2022-03-08T23:12:50.138 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:50.164 INFO:teuthology.orchestra.run.smithi107.stdout:94489280528 2022-03-08T23:12:50.181 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280528 got 94489280528 for osd.3 2022-03-08T23:12:50.181 DEBUG:teuthology.parallel:result is None 2022-03-08T23:12:50.182 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-08T23:12:50.182 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph pg dump --format=json 2022-03-08T23:12:50.620 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:50.622 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2022-03-08T23:12:50.636 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2022-03-08T23:12:49.633755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":873004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67400,"kb_used_data":6504,"kb_used_omap":0,"kb_used_meta":60800,"kb_avail":1124874424,"statfs":{"total":1151940427776,"available":1151871410176,"internally_reserved":0,"allocated":6660096,"data_stored":3834312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62259200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004378"},"pg_stats":[{"pgid":"1.0","version":"37'137","reported_seq":248,"reported_epoch":40,"state":"active+clean","last_fresh":"2022-03-08T23:12:31.228848+0000","last_change":"2022-03-08T23:12:01.824167+0000","last_active":"2022-03-08T23:12:31.228848+0000","last_peered":"2022-03-08T23:12:31.228848+0000","last_clean":"2022-03-08T23:12:31.228848+0000","last_became_active":"2022-03-08T23:12:01.823480+0000","last_became_peered":"2022-03-08T23:12:01.823480+0000","last_unstale":"2022-03-08T23:12:31.228848+0000","last_undegraded":"2022-03-08T23:12:31.228848+0000","last_fullsized":"2022-03-08T23:12:31.228848+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T23:11:43.259598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T23:11:43.259598+0000","last_clean_scrub_stamp":"2022-03-08T23:11:43.259598+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T05:55:32.496424+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":873004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":873004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2478080,"data_stored":2459788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":11,"up_from":40,"seq":171798691846,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5220,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739932,"statfs":{"total":95995035648,"available":95989690368,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.514}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.615}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.335}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8319999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7490000000000001}]}]},{"osd":10,"up_from":33,"seq":141733920780,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5284,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739868,"statfs":{"total":95995035648,"available":95989624832,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.161}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4910000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9780000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6019999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5350000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.044}]}]},{"osd":9,"up_from":33,"seq":141733920781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5996,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739156,"statfs":{"total":95995035648,"available":95988895744,"internally_reserved":0,"allocated":954368,"data_stored":716222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6050000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.8970000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1360000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6630000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.833}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4509999999999996}]}]},{"osd":6,"up_from":31,"seq":133143986189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5284,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739868,"statfs":{"total":95995035648,"available":95989624832,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6080000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]}]},{"osd":7,"up_from":29,"seq":124554051598,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5348,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6220000000000001}]}]},{"osd":4,"up_from":28,"seq":120259084303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5476,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739676,"statfs":{"total":95995035648,"available":95989428224,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8440000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4379999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]}]},{"osd":1,"up_from":20,"seq":85899345937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5348,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 8 23:12:39 2022","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.032},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.2609999999999999}]},{"osd":2,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.4790000000000001}]},{"osd":3,"last update":"Tue Mar 8 23:12:45 2022","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.5979999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.569},{"interface":"front","average":{"1min":1.403,"5min":1.403,"15min":1.403},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.343}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.845}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7729999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]}]},{"osd":8,"up_from":24,"seq":103079215119,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6508,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738644,"statfs":{"total":95995035648,"available":95988371456,"internally_reserved":0,"allocated":954368,"data_stored":716222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":1,"last update":"Tue Mar 8 23:12:46 2022","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.91000000000000003}]},{"osd":2,"last update":"Tue Mar 8 23:12:46 2022","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.645,"5min":2.645,"15min":2.645},"last":0.748},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.85599999999999998}]},{"osd":3,"last update":"Tue Mar 8 23:12:46 2022","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":1.131}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2759999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5369999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8420000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6379999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]}]},{"osd":0,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 8 23:12:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.337},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":1.7769999999999999}]},{"osd":2,"last update":"Tue Mar 8 23:12:44 2022","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":1.851},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":1.115}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.129}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.343}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]}]},{"osd":2,"up_from":21,"seq":90194313233,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6508,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738644,"statfs":{"total":95995035648,"available":95988371456,"internally_reserved":0,"allocated":954368,"data_stored":716222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":1.1950000000000001}]},{"osd":1,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":1.611},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":1.0309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8420000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5348,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":1,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.008,"15min":1.008},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.903}]},{"osd":2,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.2150000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9590000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1480000000000001}]}]},{"osd":5,"up_from":25,"seq":107374182417,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5348,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.1240000000000001}]},{"osd":1,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":1.873}]},{"osd":2,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.542}]},{"osd":3,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":1.631},{"interface":"front","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.456}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7669999999999999}]},{"osd":8,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":2.569,"5min":2.569,"15min":2.569},"last":1.97},{"interface":"front","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":1.179,"5min":1.179,"15min":1.179},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":2.2610000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3279999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T23:12:50.638 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph pg dump --format=json 2022-03-08T23:12:51.320 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:51.322 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2022-03-08T23:12:51.340 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2022-03-08T23:12:49.633755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":873004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67400,"kb_used_data":6504,"kb_used_omap":0,"kb_used_meta":60800,"kb_avail":1124874424,"statfs":{"total":1151940427776,"available":1151871410176,"internally_reserved":0,"allocated":6660096,"data_stored":3834312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62259200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004378"},"pg_stats":[{"pgid":"1.0","version":"37'137","reported_seq":248,"reported_epoch":40,"state":"active+clean","last_fresh":"2022-03-08T23:12:31.228848+0000","last_change":"2022-03-08T23:12:01.824167+0000","last_active":"2022-03-08T23:12:31.228848+0000","last_peered":"2022-03-08T23:12:31.228848+0000","last_clean":"2022-03-08T23:12:31.228848+0000","last_became_active":"2022-03-08T23:12:01.823480+0000","last_became_peered":"2022-03-08T23:12:01.823480+0000","last_unstale":"2022-03-08T23:12:31.228848+0000","last_undegraded":"2022-03-08T23:12:31.228848+0000","last_fullsized":"2022-03-08T23:12:31.228848+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T23:11:43.259598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T23:11:43.259598+0000","last_clean_scrub_stamp":"2022-03-08T23:11:43.259598+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-10T05:55:32.496424+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":873004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":873004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2478080,"data_stored":2459788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":11,"up_from":40,"seq":171798691846,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5220,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739932,"statfs":{"total":95995035648,"available":95989690368,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.514}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.615}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.335}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8319999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7490000000000001}]}]},{"osd":10,"up_from":33,"seq":141733920780,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5284,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739868,"statfs":{"total":95995035648,"available":95989624832,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.161}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4910000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9780000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6019999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5350000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.044}]}]},{"osd":9,"up_from":33,"seq":141733920781,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5996,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739156,"statfs":{"total":95995035648,"available":95988895744,"internally_reserved":0,"allocated":954368,"data_stored":716222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6050000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.8970000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.1360000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6630000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.833}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4509999999999996}]}]},{"osd":6,"up_from":31,"seq":133143986189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5284,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739868,"statfs":{"total":95995035648,"available":95989624832,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6080000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]}]},{"osd":7,"up_from":29,"seq":124554051598,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5348,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6220000000000001}]}]},{"osd":4,"up_from":28,"seq":120259084303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5476,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739676,"statfs":{"total":95995035648,"available":95989428224,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8440000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4379999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5760000000000001}]}]},{"osd":1,"up_from":20,"seq":85899345937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5348,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 8 23:12:39 2022","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.032},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.2609999999999999}]},{"osd":2,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":1.4790000000000001}]},{"osd":3,"last update":"Tue Mar 8 23:12:45 2022","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.5979999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.569},{"interface":"front","average":{"1min":1.403,"5min":1.403,"15min":1.403},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.343}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4039999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.845}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7729999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]}]},{"osd":8,"up_from":24,"seq":103079215119,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6508,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738644,"statfs":{"total":95995035648,"available":95988371456,"internally_reserved":0,"allocated":954368,"data_stored":716222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":1,"last update":"Tue Mar 8 23:12:46 2022","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.91000000000000003}]},{"osd":2,"last update":"Tue Mar 8 23:12:46 2022","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.645,"5min":2.645,"15min":2.645},"last":0.748},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.85599999999999998}]},{"osd":3,"last update":"Tue Mar 8 23:12:46 2022","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":1.131}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2759999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5369999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8420000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6379999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]}]},{"osd":0,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 8 23:12:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.337},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":1.7769999999999999}]},{"osd":2,"last update":"Tue Mar 8 23:12:44 2022","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":1.851},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":1.115}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.129}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.343}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]}]},{"osd":2,"up_from":21,"seq":90194313233,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6508,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738644,"statfs":{"total":95995035648,"available":95988371456,"internally_reserved":0,"allocated":954368,"data_stored":716222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":1.1950000000000001}]},{"osd":1,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":1.611},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":1.0309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8420000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5348,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":1,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.008,"15min":1.008},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":2.2429999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.903}]},{"osd":2,"last update":"Tue Mar 8 23:12:42 2022","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":1.2150000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9590000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1480000000000001}]}]},{"osd":5,"up_from":25,"seq":107374182417,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5348,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739804,"statfs":{"total":95995035648,"available":95989559296,"internally_reserved":0,"allocated":421888,"data_stored":187294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.1240000000000001}]},{"osd":1,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":1.873}]},{"osd":2,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.542}]},{"osd":3,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":1.631},{"interface":"front","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.456}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7669999999999999}]},{"osd":8,"last update":"Tue Mar 8 23:12:47 2022","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":2.569,"5min":2.569,"15min":2.569},"last":1.97},{"interface":"front","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":1.179,"5min":1.179,"15min":1.179},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":2.2610000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3279999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T23:12:51.343 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-08T23:12:51.343 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-08T23:12:51.344 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-08T23:12:51.344 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph health --format=json 2022-03-08T23:12:51.901 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:51.901 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-08T23:12:51.922 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-08T23:12:51.922 INFO:tasks.rook:Rook complete, yielding 2022-03-08T23:12:51.922 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:12:51.935 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph -s 2022-03-08T23:12:52.523 INFO:teuthology.orchestra.run.smithi107.stdout: cluster: 2022-03-08T23:12:52.523 INFO:teuthology.orchestra.run.smithi107.stdout: id: ed68b1de-23c1-4772-97c5-84bcc4ab6c90 2022-03-08T23:12:52.524 INFO:teuthology.orchestra.run.smithi107.stdout: health: HEALTH_OK 2022-03-08T23:12:52.524 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:52.524 INFO:teuthology.orchestra.run.smithi107.stdout: services: 2022-03-08T23:12:52.525 INFO:teuthology.orchestra.run.smithi107.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2022-03-08T23:12:52.525 INFO:teuthology.orchestra.run.smithi107.stdout: mgr: a(active, since 2m) 2022-03-08T23:12:52.525 INFO:teuthology.orchestra.run.smithi107.stdout: osd: 12 osds: 12 up (since 21s), 12 in (since 106s) 2022-03-08T23:12:52.525 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:52.526 INFO:teuthology.orchestra.run.smithi107.stdout: data: 2022-03-08T23:12:52.526 INFO:teuthology.orchestra.run.smithi107.stdout: pools: 1 pools, 1 pgs 2022-03-08T23:12:52.526 INFO:teuthology.orchestra.run.smithi107.stdout: objects: 2 objects, 577 KiB 2022-03-08T23:12:52.527 INFO:teuthology.orchestra.run.smithi107.stdout: usage: 66 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-08T23:12:52.527 INFO:teuthology.orchestra.run.smithi107.stdout: pgs: 1 active+clean 2022-03-08T23:12:52.527 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:12:52.544 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status 2022-03-08T23:12:53.291 INFO:teuthology.orchestra.run.smithi107.stdout:Backend: rook 2022-03-08T23:12:53.292 INFO:teuthology.orchestra.run.smithi107.stdout:Available: Yes 2022-03-08T23:12:53.315 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ps 2022-03-08T23:12:53.845 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:12:53.856 INFO:teuthology.orchestra.run.smithi107.stdout:crashcollector.crash smithi107 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:12:53.856 INFO:teuthology.orchestra.run.smithi107.stdout:crashcollector.crash smithi155 running (115s) 0s ago 115s - - 4b871cde5e1a 2022-03-08T23:12:53.856 INFO:teuthology.orchestra.run.smithi107.stdout:crashcollector.crash smithi187 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:12:53.857 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.a smithi155 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:12:53.857 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi187 running (6m) 0s ago 6m - - 4b871cde5e1a 2022-03-08T23:12:53.858 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi107 running (6m) 0s ago 6m - - 4b871cde5e1a 2022-03-08T23:12:53.858 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi155 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T23:12:53.858 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi155 running (115s) 0s ago 115s - - 4b871cde5e1a 2022-03-08T23:12:53.858 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi187 running (113s) 0s ago 114s - - 4b871cde5e1a 2022-03-08T23:12:53.859 INFO:teuthology.orchestra.run.smithi107.stdout:osd.10 smithi107 running (99s) 0s ago 99s - - 4b871cde5e1a 2022-03-08T23:12:53.859 INFO:teuthology.orchestra.run.smithi107.stdout:osd.11 smithi155 running (43s) 0s ago 43s - - 4b871cde5e1a 2022-03-08T23:12:53.859 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi187 running (111s) 0s ago 111s - - 4b871cde5e1a 2022-03-08T23:12:53.860 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi187 running (111s) 0s ago 111s - - 4b871cde5e1a 2022-03-08T23:12:53.860 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi107 running (108s) 0s ago 108s - - 4b871cde5e1a 2022-03-08T23:12:53.860 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi155 running (105s) 0s ago 105s - - 4b871cde5e1a 2022-03-08T23:12:53.860 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi107 running (106s) 0s ago 107s - - 4b871cde5e1a 2022-03-08T23:12:53.861 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi155 running (102s) 0s ago 102s - - 4b871cde5e1a 2022-03-08T23:12:53.861 INFO:teuthology.orchestra.run.smithi107.stdout:osd.8 smithi187 running (103s) 0s ago 103s - - 4b871cde5e1a 2022-03-08T23:12:53.861 INFO:teuthology.orchestra.run.smithi107.stdout:osd.9 smithi107 running (101s) 0s ago 101s - - 4b871cde5e1a 2022-03-08T23:12:53.869 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls 2022-03-08T23:12:54.537 INFO:teuthology.orchestra.run.smithi107.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:12:54.538 INFO:teuthology.orchestra.run.smithi107.stdout:crash 3/3 0s ago 3m * 2022-03-08T23:12:54.538 INFO:teuthology.orchestra.run.smithi107.stdout:mgr 1/1 0s ago 3m count:1 2022-03-08T23:12:54.538 INFO:teuthology.orchestra.run.smithi107.stdout:mon 3/3 0s ago 6m count:3 2022-03-08T23:12:54.538 INFO:teuthology.orchestra.run.smithi107.stdout:osd 24 0s ago 116s 2022-03-08T23:12:54.539 INFO:teuthology.orchestra.run.smithi107.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T23:12:54.562 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch device ls 2022-03-08T23:12:55.098 INFO:teuthology.orchestra.run.smithi107.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:12:55.098 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:12:55.099 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:12:55.099 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:12:55.100 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:12:55.100 INFO:teuthology.orchestra.run.smithi107.stdout:smithi155 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:12:55.100 INFO:teuthology.orchestra.run.smithi107.stdout:smithi155 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:12:55.100 INFO:teuthology.orchestra.run.smithi107.stdout:smithi155 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:12:55.100 INFO:teuthology.orchestra.run.smithi107.stdout:smithi155 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:12:55.101 INFO:teuthology.orchestra.run.smithi107.stdout:smithi187 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:12:55.101 INFO:teuthology.orchestra.run.smithi107.stdout:smithi187 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:12:55.101 INFO:teuthology.orchestra.run.smithi107.stdout:smithi187 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:12:55.101 INFO:teuthology.orchestra.run.smithi107.stdout:smithi187 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:12:55.122 INFO:teuthology.run_tasks:Running task exec... 2022-03-08T23:12:55.135 INFO:teuthology.task.exec:Executing custom commands... 2022-03-08T23:12:55.135 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi107.front.sepia.ceph.com 2022-03-08T23:12:55.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-08T23:12:55.136 DEBUG:teuthology.orchestra.run.smithi107:> toolbox() { 2022-03-08T23:12:55.136 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-08T23:12:55.136 DEBUG:teuthology.orchestra.run.smithi107:> } 2022-03-08T23:12:55.137 DEBUG:teuthology.orchestra.run.smithi107:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:12:55.137 DEBUG:teuthology.orchestra.run.smithi107:> toolbox ceph orch osd rm 0 --force 2022-03-08T23:12:55.138 DEBUG:teuthology.orchestra.run.smithi107:> removed_pv="" 2022-03-08T23:12:55.138 DEBUG:teuthology.orchestra.run.smithi107:> while [ "$removed_pv" = "" ] 2022-03-08T23:12:55.138 DEBUG:teuthology.orchestra.run.smithi107:> do 2022-03-08T23:12:55.139 DEBUG:teuthology.orchestra.run.smithi107:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-08T23:12:55.139 DEBUG:teuthology.orchestra.run.smithi107:> sleep 3s 2022-03-08T23:12:55.139 DEBUG:teuthology.orchestra.run.smithi107:> done 2022-03-08T23:12:55.139 DEBUG:teuthology.orchestra.run.smithi107:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-08T23:12:55.140 DEBUG:teuthology.orchestra.run.smithi107:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-08T23:12:55.140 DEBUG:teuthology.orchestra.run.smithi107:> toolbox ceph orch device zap $host $target_path --force 2022-03-08T23:12:55.140 DEBUG:teuthology.orchestra.run.smithi107:> zap_completion="0" 2022-03-08T23:12:55.141 DEBUG:teuthology.orchestra.run.smithi107:> while [ "$zap_completion" = "0" ] 2022-03-08T23:12:55.141 DEBUG:teuthology.orchestra.run.smithi107:> do 2022-03-08T23:12:55.142 DEBUG:teuthology.orchestra.run.smithi107:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-08T23:12:55.142 DEBUG:teuthology.orchestra.run.smithi107:> sleep 3s 2022-03-08T23:12:55.142 DEBUG:teuthology.orchestra.run.smithi107:> done 2022-03-08T23:12:55.142 DEBUG:teuthology.orchestra.run.smithi107:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-08T23:12:55.143 DEBUG:teuthology.orchestra.run.smithi107:> toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:12:55.143 DEBUG:teuthology.orchestra.run.smithi107:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:12:55.143 DEBUG:teuthology.orchestra.run.smithi107:> num_osd="0" 2022-03-08T23:12:55.144 DEBUG:teuthology.orchestra.run.smithi107:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-08T23:12:55.144 DEBUG:teuthology.orchestra.run.smithi107:> do 2022-03-08T23:12:55.144 DEBUG:teuthology.orchestra.run.smithi107:> echo "waiting for osd to come back up" 2022-03-08T23:12:55.144 DEBUG:teuthology.orchestra.run.smithi107:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T23:12:55.145 DEBUG:teuthology.orchestra.run.smithi107:> sleep 30s 2022-03-08T23:12:55.145 DEBUG:teuthology.orchestra.run.smithi107:> done 2022-03-08T23:12:55.145 DEBUG:teuthology.orchestra.run.smithi107:> ' 2022-03-08T23:12:55.161 INFO:teuthology.orchestra.run.smithi107.stderr:++ toolbox ceph osd stat 2022-03-08T23:12:55.161 INFO:teuthology.orchestra.run.smithi107.stderr:++ cut -f3 -d ' ' 2022-03-08T23:12:55.162 INFO:teuthology.orchestra.run.smithi107.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:12:57.922 INFO:teuthology.orchestra.run.smithi107.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:12:58.361 INFO:teuthology.orchestra.run.smithi107.stderr:+ orig_num_osd=12 2022-03-08T23:12:58.362 INFO:teuthology.orchestra.run.smithi107.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-08T23:12:58.362 INFO:teuthology.orchestra.run.smithi107.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-08T23:12:58.417 INFO:teuthology.orchestra.run.smithi107.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:12:59.057 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.053878 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:12:59.057 INFO:tasks.rook.operator.smithi107.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T23:12:59.058 INFO:tasks.rook.operator.smithi107.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:12:59.058 INFO:tasks.rook.operator.smithi107.stdout:   ... // 3 identical fields 2022-03-08T23:12:59.058 INFO:tasks.rook.operator.smithi107.stdout:   Config: nil, 2022-03-08T23:12:59.059 INFO:tasks.rook.operator.smithi107.stdout:   Selection: {}, 2022-03-08T23:12:59.059 INFO:tasks.rook.operator.smithi107.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:12:59.059 INFO:tasks.rook.operator.smithi107.stdout:   ... // 3 identical elements 2022-03-08T23:12:59.060 INFO:tasks.rook.operator.smithi107.stdout:   {Name: "smithi107-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi107-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:12:59.060 INFO:tasks.rook.operator.smithi107.stdout:   {Name: "smithi155-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi155-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:12:59.060 INFO:tasks.rook.operator.smithi107.stdout:-  { 2022-03-08T23:12:59.060 INFO:tasks.rook.operator.smithi107.stdout:-  Name: "smithi155-lv-2", 2022-03-08T23:12:59.061 INFO:tasks.rook.operator.smithi107.stdout:-  Count: 1, 2022-03-08T23:12:59.061 INFO:tasks.rook.operator.smithi107.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:12:59.061 INFO:tasks.rook.operator.smithi107.stdout:-  }, 2022-03-08T23:12:59.061 INFO:tasks.rook.operator.smithi107.stdout:   {Name: "smithi155-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi155-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:12:59.061 INFO:tasks.rook.operator.smithi107.stdout:   {Name: "smithi155-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi155-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:12:59.062 INFO:tasks.rook.operator.smithi107.stdout:   ... // 4 identical elements 2022-03-08T23:12:59.062 INFO:tasks.rook.operator.smithi107.stdout:   }, 2022-03-08T23:12:59.062 INFO:tasks.rook.operator.smithi107.stdout:   }, 2022-03-08T23:12:59.062 INFO:tasks.rook.operator.smithi107.stdout:   Annotations: nil, 2022-03-08T23:12:59.063 INFO:tasks.rook.operator.smithi107.stdout:   Labels: nil, 2022-03-08T23:12:59.063 INFO:tasks.rook.operator.smithi107.stdout:   ... // 20 identical fields 2022-03-08T23:12:59.063 INFO:tasks.rook.operator.smithi107.stdout:  } 2022-03-08T23:12:59.063 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.053986 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:12:59.063 INFO:tasks.rook.operator.smithi107.stdout:I0308 23:12:59.054218 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:12:59.064 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.054932 I | operator: successfully started the controller-runtime manager 2022-03-08T23:12:59.090 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.088393 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:12:59.091 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.088428 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:12:59.092 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.088464 I | operator: setting up schemes 2022-03-08T23:12:59.095 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.092959 I | operator: setting up the controller-runtime manager 2022-03-08T23:12:59.398 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.397081 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:12:59.420 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.418419 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:12:59.420 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.418511 I | ceph-cluster-controller: successfully started 2022-03-08T23:12:59.421 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.418615 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:12:59.423 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.418782 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:12:59.424 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.418861 I | ceph-block-pool-controller: successfully started 2022-03-08T23:12:59.424 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.418919 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:12:59.424 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.418970 I | ceph-object-realm-controller: successfully started 2022-03-08T23:12:59.425 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.418996 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:12:59.426 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419032 I | ceph-object-zone-controller: successfully started 2022-03-08T23:12:59.426 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419391 I | ceph-object-controller: successfully started 2022-03-08T23:12:59.426 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419479 I | ceph-file-controller: successfully started 2022-03-08T23:12:59.426 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419587 I | ceph-nfs-controller: successfully started 2022-03-08T23:12:59.427 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419650 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:12:59.427 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419696 I | ceph-client-controller: successfully started 2022-03-08T23:12:59.427 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419733 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:12:59.427 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419786 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:12:59.428 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419819 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:12:59.428 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419857 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:12:59.428 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419956 I | ceph-bucket-topic: successfully started 2022-03-08T23:12:59.429 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.419995 I | ceph-bucket-notification: successfully started 2022-03-08T23:12:59.429 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.420013 I | ceph-bucket-notification: successfully started 2022-03-08T23:12:59.429 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.420030 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:12:59.429 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.421569 I | operator: starting the controller-runtime manager 2022-03-08T23:12:59.533 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.531499 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:12:59.534 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.531567 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:12:59.534 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.531598 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:12:59.539 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.532236 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:12:59.570 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.569093 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:12:59.575 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.571356 I | op-mon: parsing mon endpoints: c=10.97.212.176:6789,a=10.99.149.173:6789,b=10.100.13.26:6789 2022-03-08T23:12:59.575 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.571866 I | op-mon: parsing mon endpoints: c=10.97.212.176:6789,a=10.99.149.173:6789,b=10.100.13.26:6789 2022-03-08T23:12:59.575 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.571930 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:12:59.576 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.571942 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:12:59.579 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.574295 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:12:59.579 INFO:tasks.rook.operator.smithi107.stdout:I0308 23:12:59.574517 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:12:59.640 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.637598 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:12:59.644 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.640626 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:12:59.712 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.708991 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:12:59.713 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709016 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:12:59.713 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709024 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:12:59.714 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709030 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:12:59.714 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709035 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:12:59.714 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709039 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:12:59.715 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709045 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:12:59.715 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709050 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:12:59.715 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709056 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:12:59.715 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709061 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:12:59.716 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709066 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:12:59.716 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709072 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:12:59.716 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709078 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:12:59.716 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709082 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:12:59.716 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709087 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:12:59.717 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709094 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:12:59.717 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709137 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:12:59.717 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:12:59.709145 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:13:03.627 INFO:teuthology.orchestra.run.smithi107.stdout:Success 2022-03-08T23:13:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:+ removed_pv= 2022-03-08T23:13:03.649 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' '' = '' ']' 2022-03-08T23:13:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:++ kubectl get pv 2022-03-08T23:13:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:++ grep Released 2022-03-08T23:13:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:++ cut -f1 -d ' ' 2022-03-08T23:13:04.114 INFO:teuthology.orchestra.run.smithi107.stderr:+ removed_pv= 2022-03-08T23:13:04.114 INFO:teuthology.orchestra.run.smithi107.stderr:+ sleep 3s 2022-03-08T23:13:04.645 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:04.643859 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-08T23:13:07.115 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' '' = '' ']' 2022-03-08T23:13:07.116 INFO:teuthology.orchestra.run.smithi107.stderr:++ kubectl get pv 2022-03-08T23:13:07.116 INFO:teuthology.orchestra.run.smithi107.stderr:++ grep Released 2022-03-08T23:13:07.116 INFO:teuthology.orchestra.run.smithi107.stderr:++ cut -f1 -d ' ' 2022-03-08T23:13:07.181 INFO:teuthology.orchestra.run.smithi107.stderr:+ removed_pv=smithi155-lv-2 2022-03-08T23:13:07.182 INFO:teuthology.orchestra.run.smithi107.stderr:+ sleep 3s 2022-03-08T23:13:08.498 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:08.496425 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:13:08.499 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:08.496450 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:13:08.499 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:08.496458 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:13:08.629 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:08.627035 I | op-mon: parsing mon endpoints: c=10.97.212.176:6789,a=10.99.149.173:6789,b=10.100.13.26:6789 2022-03-08T23:13:08.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:08.692788 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:13:08.696 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:08.692990 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:13:09.180 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.179285 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:13:09.349 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342869 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:13:09.349 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342898 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:13:09.350 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342908 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:13:09.350 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342912 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:13:09.351 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342917 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:13:09.351 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342921 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:13:09.351 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342927 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:13:09.352 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342931 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:13:09.352 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342937 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:13:09.352 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342942 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:13:09.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342951 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:13:09.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342956 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:13:09.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342960 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:13:09.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342965 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:13:09.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342969 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:13:09.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342975 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:13:09.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342983 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:13:09.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342989 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:13:09.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.342994 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:13:09.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.343000 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:13:09.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.343005 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:13:09.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.343010 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:13:09.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.343015 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:13:09.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.343022 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:13:09.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.343026 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:13:09.722 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.720379 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:13:09.730 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.727742 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:13:09.730 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.727764 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:13:09.731 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.727768 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:13:09.731 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.727772 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:13:09.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.727775 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:13:09.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.727777 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:13:09.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:09.727781 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:13:10.182 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' smithi155-lv-2 = '' ']' 2022-03-08T23:13:10.183 INFO:teuthology.orchestra.run.smithi107.stderr:++ kubectl get pv smithi155-lv-2 -o 'jsonpath={.spec.local.path}' 2022-03-08T23:13:10.233 INFO:teuthology.orchestra.run.smithi107.stderr:+ target_path=/dev/vg_nvme/lv_2 2022-03-08T23:13:10.233 INFO:teuthology.orchestra.run.smithi107.stderr:++ echo smithi155-lv-2 2022-03-08T23:13:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:++ cut -f1 -d - 2022-03-08T23:13:10.236 INFO:teuthology.orchestra.run.smithi107.stderr:+ host=smithi155 2022-03-08T23:13:10.237 INFO:teuthology.orchestra.run.smithi107.stderr:+ toolbox ceph orch device zap smithi155 /dev/vg_nvme/lv_2 --force 2022-03-08T23:13:10.237 INFO:teuthology.orchestra.run.smithi107.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi155 /dev/vg_nvme/lv_2 --force 2022-03-08T23:13:10.267 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:10.264517 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:13:10.268 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:10.264553 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:13:10.268 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:10.264564 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:13:10.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:10.266094 I | op-mon: start running mons 2022-03-08T23:13:10.376 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:10.370967 I | op-mon: parsing mon endpoints: c=10.97.212.176:6789,a=10.99.149.173:6789,b=10.100.13.26:6789 2022-03-08T23:13:10.436 INFO:teuthology.orchestra.run.smithi107.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:13:10.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:10.453406 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:13:10.553 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:10.550664 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.212.176:6789","10.99.149.173:6789","10.100.13.26:6789"]}] data:b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:13:10.714 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:10.713000 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:13:10.715 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:10.713234 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:13:10.982 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/vg_nvme/lv_2 on smithi155 zapped 2022-03-08T23:13:11.002 INFO:teuthology.orchestra.run.smithi107.stderr:+ zap_completion=0 2022-03-08T23:13:11.002 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' 0 = 0 ']' 2022-03-08T23:13:11.003 INFO:teuthology.orchestra.run.smithi107.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-08T23:13:11.112 INFO:teuthology.orchestra.run.smithi107.stderr:+ zap_completion= 2022-03-08T23:13:11.112 INFO:teuthology.orchestra.run.smithi107.stderr:+ sleep 3s 2022-03-08T23:13:11.166 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:11.160656 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:13:11.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:11.160696 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:13:11.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:11.160710 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:13:11.751 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:11.749816 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:13:11.752 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:11.749846 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:13:11.752 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:11.749854 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:13:12.113 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:12.111447 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:13:12.938 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:12.936406 I | op-mon: targeting the mon count 3 2022-03-08T23:13:12.944 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:12.942757 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:13:13.141 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:13.138991 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:13:13.146 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:13.144339 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:13:13.146 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:13.144353 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:13:13.151 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:13.148726 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:13:13.305 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:13.303486 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:13:13.306 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:13.303527 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:13:13.653 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:13.651077 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:13:13.653 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:13.651108 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:13:14.113 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' '' = 0 ']' 2022-03-08T23:13:14.114 INFO:teuthology.orchestra.run.smithi107.stderr:+ kubectl patch pv smithi155-lv-2 -p '{"spec":{"claimRef": null}}' 2022-03-08T23:13:14.175 INFO:teuthology.orchestra.run.smithi107.stdout:persistentvolume/smithi155-lv-2 patched 2022-03-08T23:13:14.177 INFO:teuthology.orchestra.run.smithi107.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-08T23:13:14.177 INFO:teuthology.orchestra.run.smithi107.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-08T23:13:14.196 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.192649 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:13:14.196 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.192680 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:13:14.278 INFO:teuthology.orchestra.run.smithi107.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:13:14.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.637663 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:13:14.640 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.637691 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:13:14.899 INFO:teuthology.orchestra.run.smithi107.stdout:['Success'] 2022-03-08T23:13:14.904 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.901392 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-08T23:13:14.904 INFO:tasks.rook.operator.smithi107.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T23:13:14.904 INFO:tasks.rook.operator.smithi107.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-08T23:13:14.905 INFO:tasks.rook.operator.smithi107.stdout:   ... // 3 identical fields 2022-03-08T23:13:14.905 INFO:tasks.rook.operator.smithi107.stdout:   Config: nil, 2022-03-08T23:13:14.905 INFO:tasks.rook.operator.smithi107.stdout:   Selection: {}, 2022-03-08T23:13:14.906 INFO:tasks.rook.operator.smithi107.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T23:13:14.906 INFO:tasks.rook.operator.smithi107.stdout:   ... // 9 identical elements 2022-03-08T23:13:14.906 INFO:tasks.rook.operator.smithi107.stdout:   {Name: "smithi187-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi187-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:13:14.906 INFO:tasks.rook.operator.smithi107.stdout:   {Name: "smithi187-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi187-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T23:13:14.907 INFO:tasks.rook.operator.smithi107.stdout:+  { 2022-03-08T23:13:14.907 INFO:tasks.rook.operator.smithi107.stdout:+  Name: "smithi155-lv-2", 2022-03-08T23:13:14.907 INFO:tasks.rook.operator.smithi107.stdout:+  Count: 1, 2022-03-08T23:13:14.907 INFO:tasks.rook.operator.smithi107.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T23:13:14.908 INFO:tasks.rook.operator.smithi107.stdout:+  }, 2022-03-08T23:13:14.908 INFO:tasks.rook.operator.smithi107.stdout:   }, 2022-03-08T23:13:14.908 INFO:tasks.rook.operator.smithi107.stdout:   }, 2022-03-08T23:13:14.908 INFO:tasks.rook.operator.smithi107.stdout:   Annotations: nil, 2022-03-08T23:13:14.909 INFO:tasks.rook.operator.smithi107.stdout:   Labels: nil, 2022-03-08T23:13:14.909 INFO:tasks.rook.operator.smithi107.stdout:   ... // 20 identical fields 2022-03-08T23:13:14.909 INFO:tasks.rook.operator.smithi107.stdout:  } 2022-03-08T23:13:14.910 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.901470 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:13:14.910 INFO:tasks.rook.operator.smithi107.stdout:I0308 23:13:14.901658 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:13:14.921 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.907700 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:13:14.921 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.907725 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:13:14.921 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.907754 I | operator: setting up schemes 2022-03-08T23:13:14.922 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:14.910254 I | operator: setting up the controller-runtime manager 2022-03-08T23:13:14.932 INFO:teuthology.orchestra.run.smithi107.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T23:13:15.026 INFO:teuthology.orchestra.run.smithi107.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-08T23:13:15.034 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.033117 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:13:15.035 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.033136 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:13:15.039 INFO:teuthology.orchestra.run.smithi107.stderr:+ num_osd=0 2022-03-08T23:13:15.039 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' 0 '!=' 12 ']' 2022-03-08T23:13:15.039 INFO:teuthology.orchestra.run.smithi107.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:13:15.040 INFO:teuthology.orchestra.run.smithi107.stdout:waiting for osd to come back up 2022-03-08T23:13:15.040 INFO:teuthology.orchestra.run.smithi107.stderr:++ toolbox ceph osd stat 2022-03-08T23:13:15.041 INFO:teuthology.orchestra.run.smithi107.stderr:++ cut -f3 -d ' ' 2022-03-08T23:13:15.041 INFO:teuthology.orchestra.run.smithi107.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:13:15.049 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.047736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-03-08T23:13:15.049 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.047844 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:13:15.104 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.102249 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:13:15.119 INFO:teuthology.orchestra.run.smithi107.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:13:15.123 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.122132 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-03-08 23:13:15.115202985 +0000 UTC m=+485.653521389 LastTransitionTime:2022-03-08 23:13:15.115202856 +0000 UTC m=+485.653521286}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:13:15.124 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.122156 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:13:15.215 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.213521 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:13:15.258 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.255827 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:13:15.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.255866 I | ceph-cluster-controller: successfully started 2022-03-08T23:13:15.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.255918 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:13:15.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.255938 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:13:15.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.255958 I | ceph-block-pool-controller: successfully started 2022-03-08T23:13:15.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.255978 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:13:15.260 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256000 I | ceph-object-realm-controller: successfully started 2022-03-08T23:13:15.260 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256012 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:13:15.260 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256022 I | ceph-object-zone-controller: successfully started 2022-03-08T23:13:15.268 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256174 I | ceph-object-controller: successfully started 2022-03-08T23:13:15.268 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256212 I | ceph-file-controller: successfully started 2022-03-08T23:13:15.268 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256247 I | ceph-nfs-controller: successfully started 2022-03-08T23:13:15.268 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256272 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:13:15.269 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256299 I | ceph-client-controller: successfully started 2022-03-08T23:13:15.269 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256317 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:13:15.269 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256340 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:13:15.269 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256357 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:13:15.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256372 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:13:15.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256386 I | ceph-bucket-topic: successfully started 2022-03-08T23:13:15.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256400 I | ceph-bucket-notification: successfully started 2022-03-08T23:13:15.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256411 I | ceph-bucket-notification: successfully started 2022-03-08T23:13:15.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.256422 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:13:15.271 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.257532 I | operator: starting the controller-runtime manager 2022-03-08T23:13:15.365 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.362722 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:13:15.368 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.362777 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:13:15.369 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.362826 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:13:15.369 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.362843 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:13:15.373 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.367428 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:13:15.379 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.368572 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:13:15.379 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.368621 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T23:13:15.380 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.368630 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:13:15.380 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.369942 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:13:15.380 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.370262 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T23:13:15.380 INFO:tasks.rook.operator.smithi107.stdout:I0308 23:13:15.370320 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T23:13:15.423 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.421540 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:13:15.545 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.543651 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T23:13:15.664 INFO:teuthology.orchestra.run.smithi107.stderr:+ num_osd=11 2022-03-08T23:13:15.665 INFO:teuthology.orchestra.run.smithi107.stderr:+ sleep 30s 2022-03-08T23:13:15.752 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.750932 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T23:13:15.784 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.782156 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:13:15.945 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941510 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T23:13:15.945 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941539 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T23:13:15.945 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941545 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T23:13:15.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941550 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T23:13:15.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941556 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T23:13:15.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941561 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T23:13:15.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941566 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T23:13:15.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941572 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T23:13:15.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941577 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T23:13:15.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941582 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T23:13:15.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941587 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T23:13:15.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941593 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T23:13:15.949 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941599 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T23:13:15.949 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941602 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T23:13:15.949 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941607 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T23:13:15.950 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941613 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T23:13:15.950 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941676 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:13:15.950 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:15.941689 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:13:18.791 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:18.789787 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:13:21.819 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:21.817345 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T23:13:22.240 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236868 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T23:13:22.241 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236914 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T23:13:22.241 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236928 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:13:22.241 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236935 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T23:13:22.241 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236942 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:13:22.242 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236948 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T23:13:22.242 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236954 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:13:22.242 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236960 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T23:13:22.242 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236967 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:13:22.243 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236972 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T23:13:22.243 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236982 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:13:22.243 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.236992 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T23:13:22.243 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237003 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:13:22.243 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237013 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T23:13:22.244 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237023 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T23:13:22.244 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237034 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T23:13:22.244 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237045 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T23:13:22.244 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237057 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T23:13:22.244 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237069 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T23:13:22.245 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237080 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:13:22.245 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237097 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T23:13:22.246 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237108 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T23:13:22.247 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237120 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T23:13:22.247 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237133 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T23:13:22.247 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.237143 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T23:13:22.264 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.262786 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T23:13:22.274 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.272753 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:13:22.275 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.272841 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:13:22.275 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.272856 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:13:22.275 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.272862 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:13:22.275 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.272880 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:13:22.276 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.272899 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:13:22.276 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.272908 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T23:13:22.446 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.444700 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:13:22.447 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.444729 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:13:22.447 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.444742 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:13:22.524 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.522190 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T23:13:22.785 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.783566 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T23:13:22.786 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.783592 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T23:13:22.786 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.783599 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T23:13:22.882 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.879959 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T23:13:22.882 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.879976 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T23:13:22.882 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.879980 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T23:13:22.941 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:22.939705 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T23:13:23.233 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:23.231192 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:13:23.238 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:23.236747 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T23:13:23.238 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:23.236771 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T23:13:23.245 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:23.241597 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T23:13:24.824 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:24.822513 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T23:13:30.554 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.551667 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:13:30.554 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.551697 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:13:30.555 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.551719 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:13:30.597 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.595266 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:13:30.598 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.595292 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:13:30.598 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.595305 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:13:30.603 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.598565 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:13:30.689 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.687142 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:13:30.699 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.697207 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:13:30.699 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.697233 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:13:30.699 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.697481 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:13:30.700 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:30.697496 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:13:31.191 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:31.189751 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:13:31.251 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:31.249794 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:13:31.282 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:31.280823 I | op-mon: start running mons 2022-03-08T23:13:31.340 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:31.338790 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:13:31.344 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:31.342769 E | ceph-cluster-controller: failed to update cluster "rook-ceph" version. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:13:32.222 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:32.220708 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.13.26:6789","10.97.212.176:6789","10.99.149.173:6789"]}] data:b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:13:32.821 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:32.819876 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:13:32.822 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:32.820103 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:13:34.422 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:34.420158 I | op-mon: targeting the mon count 3 2022-03-08T23:13:34.429 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:34.428055 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:13:34.778 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:34.775619 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:13:34.779 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:34.775649 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:13:35.128 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:35.126291 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:13:35.128 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:35.126320 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:13:35.570 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:35.568361 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:13:35.570 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:35.568383 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:13:35.907 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:35.906142 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:13:35.908 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:35.906175 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:13:36.236 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:36.234522 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:13:36.236 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:36.234549 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:13:36.342 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:36.340734 I | op-mon: mon "b" endpoint is [v2:10.100.13.26:3300,v1:10.100.13.26:6789] 2022-03-08T23:13:36.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:36.492259 I | op-mon: mon "c" endpoint is [v2:10.97.212.176:3300,v1:10.97.212.176:6789] 2022-03-08T23:13:36.823 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:36.821375 I | op-mon: mon "a" endpoint is [v2:10.99.149.173:3300,v1:10.99.149.173:6789] 2022-03-08T23:13:37.423 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:37.422171 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.13.26:6789","10.97.212.176:6789","10.99.149.173:6789"]}] data:b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:13:38.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:38.021230 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:13:38.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:38.021574 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:13:38.440 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:38.438842 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:13:38.453 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:38.451725 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:13:38.454 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:38.451754 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:13:39.031 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:39.029395 I | op-mon: mons running: [b c a] 2022-03-08T23:13:39.439 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:39.437195 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:13:39.444 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:39.443392 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:13:39.452 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:39.451509 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:13:39.453 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:39.451532 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:13:39.632 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:39.630473 I | op-mon: mons running: [b c a] 2022-03-08T23:13:40.092 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:40.090442 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:13:40.101 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:40.099450 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:13:40.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:40.110128 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:13:40.116 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:40.110791 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:13:40.231 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:40.229871 I | op-mon: mons running: [b c a] 2022-03-08T23:13:40.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:40.637364 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:13:40.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:40.637394 I | op-mon: mons created: 3 2022-03-08T23:13:41.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:41.052883 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:13:41.084 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:41.082956 I | op-mon: mons running: [b c a] 2022-03-08T23:13:41.560 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:41.558713 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:13:41.561 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:41.558750 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:13:41.561 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:41.558831 I | op-mon: start running mons 2022-03-08T23:13:41.566 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:41.565220 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:13:41.978 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:41.976251 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:13:42.223 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:42.221897 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789","10.100.13.26:6789","10.97.212.176:6789"]}] data:b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:13:42.401 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:42.399078 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:13:42.821 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:42.819725 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:13:42.825 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:42.824253 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:13:43.146 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:43.144476 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:13:45.223 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:45.221192 I | op-mon: targeting the mon count 3 2022-03-08T23:13:45.230 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:45.229128 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:13:45.585 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:45.583507 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:13:45.585 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:45.583538 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:13:45.665 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T23:13:45.666 INFO:teuthology.orchestra.run.smithi107.stdout:waiting for osd to come back up 2022-03-08T23:13:45.667 INFO:teuthology.orchestra.run.smithi107.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:13:45.667 INFO:teuthology.orchestra.run.smithi107.stderr:++ toolbox ceph osd stat 2022-03-08T23:13:45.667 INFO:teuthology.orchestra.run.smithi107.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:13:45.668 INFO:teuthology.orchestra.run.smithi107.stderr:++ cut -f3 -d ' ' 2022-03-08T23:13:45.733 INFO:teuthology.orchestra.run.smithi107.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:13:45.926 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:45.924783 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:13:45.927 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:45.924817 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:13:46.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.021616 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:13:46.024 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.021648 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:13:46.182 INFO:teuthology.orchestra.run.smithi107.stderr:+ num_osd=11 2022-03-08T23:13:46.182 INFO:teuthology.orchestra.run.smithi107.stderr:+ sleep 30s 2022-03-08T23:13:46.276 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.274109 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:13:46.276 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.274139 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:13:46.447 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.445681 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:13:46.623 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.621229 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:13:46.623 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.621257 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:13:46.861 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.859470 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:13:46.862 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.859509 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:13:46.862 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.859519 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:13:46.970 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.969189 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:13:46.971 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:46.969224 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:13:47.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:47.258001 I | op-mon: mon "b" endpoint is [v2:10.100.13.26:3300,v1:10.100.13.26:6789] 2022-03-08T23:13:47.323 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:47.322053 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:13:48.036 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:48.034763 I | op-mgr: start running mgr 2022-03-08T23:13:48.037 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:48.034812 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:13:48.423 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:48.421664 I | op-mon: mon "c" endpoint is [v2:10.97.212.176:3300,v1:10.97.212.176:6789] 2022-03-08T23:13:49.654 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:49.652295 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:13:49.664 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:49.662798 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:13:49.667 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:49.666169 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:13:49.825 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:49.823150 I | op-mon: mon "a" endpoint is [v2:10.99.149.173:3300,v1:10.99.149.173:6789] 2022-03-08T23:13:51.024 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:51.021760 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.149.173:6789","10.100.13.26:6789","10.97.212.176:6789"]}] data:b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:13:51.626 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:51.624593 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:13:51.627 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:51.624678 I | op-mgr: successful modules: balancer 2022-03-08T23:13:51.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:51.636714 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:13:51.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:51.636741 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:13:51.708 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:51.706281 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:13:52.022 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020535 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-1" 2022-03-08T23:13:52.022 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020572 I | op-osd: OSD PVC "smithi107-lv-1-data-0mjtjg" already exists 2022-03-08T23:13:52.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020583 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-2" 2022-03-08T23:13:52.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020594 I | op-osd: OSD PVC "smithi107-lv-2-data-09kfs9" already exists 2022-03-08T23:13:52.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020619 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-3" 2022-03-08T23:13:52.026 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020641 I | op-osd: OSD PVC "smithi107-lv-3-data-0r8x5p" already exists 2022-03-08T23:13:52.026 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020659 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-4" 2022-03-08T23:13:52.026 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020671 I | op-osd: OSD PVC "smithi107-lv-4-data-0mptlv" already exists 2022-03-08T23:13:52.027 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020679 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-1" 2022-03-08T23:13:52.027 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020695 I | op-osd: OSD PVC "smithi155-lv-1-data-0c6ssv" already exists 2022-03-08T23:13:52.027 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020703 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-3" 2022-03-08T23:13:52.027 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020712 I | op-osd: OSD PVC "smithi155-lv-3-data-08gx6j" already exists 2022-03-08T23:13:52.028 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020717 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-4" 2022-03-08T23:13:52.028 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020725 I | op-osd: OSD PVC "smithi155-lv-4-data-04c4td" already exists 2022-03-08T23:13:52.028 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020730 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-1" 2022-03-08T23:13:52.028 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020738 I | op-osd: OSD PVC "smithi187-lv-1-data-067hpz" already exists 2022-03-08T23:13:52.029 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020744 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-2" 2022-03-08T23:13:52.029 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020751 I | op-osd: OSD PVC "smithi187-lv-2-data-0wdspq" already exists 2022-03-08T23:13:52.029 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020762 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-3" 2022-03-08T23:13:52.029 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020774 I | op-osd: OSD PVC "smithi187-lv-3-data-0zsp69" already exists 2022-03-08T23:13:52.029 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020780 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-4" 2022-03-08T23:13:52.030 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020788 I | op-osd: OSD PVC "smithi187-lv-4-data-069kzv" already exists 2022-03-08T23:13:52.030 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.020794 I | op-osd: creating 1 new PVCs for device set "smithi155-lv-2" 2022-03-08T23:13:52.222 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.220799 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:13:52.227 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.221099 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:13:52.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.452856 I | op-osd: successfully provisioned PVC "smithi155-lv-2-data-0w82ls" 2022-03-08T23:13:52.477 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.475753 I | op-mgr: successful modules: prometheus 2022-03-08T23:13:52.481 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.479606 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:13:52.483 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.482113 I | op-mgr: successful modules: dashboard 2022-03-08T23:13:52.489 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484260 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:13:52.489 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484274 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-1-data-0mjtjg" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.489 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484277 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:13:52.489 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484281 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-2-data-09kfs9" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.490 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484283 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:13:52.490 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484287 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-3-data-0r8x5p" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.490 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484291 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:13:52.490 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484295 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-4-data-0mptlv" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484300 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:13:52.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484304 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-1-data-0c6ssv" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484309 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:13:52.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484315 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-3-data-08gx6j" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484320 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-4-data-04c4td" 2022-03-08T23:13:52.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484325 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-4-data-04c4td" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484330 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:13:52.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484335 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-1-data-067hpz" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484341 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:13:52.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484345 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-2-data-0wdspq" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484356 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:13:52.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484359 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-3-data-0zsp69" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484363 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:13:52.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484368 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-4-data-069kzv" because OSD daemon using the PVC already exists 2022-03-08T23:13:52.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:52.484374 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0w82ls" 2022-03-08T23:13:53.065 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.063734 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:13:53.076 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.074941 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:13:53.077 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.074967 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:13:53.243 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.241608 I | op-osd: started OSD provisioning job for PVC "smithi155-lv-2-data-0w82ls" 2022-03-08T23:13:53.244 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.241636 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:13:53.244 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.241644 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:13:53.622 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.620579 I | op-osd: OSD orchestration status for PVC smithi155-lv-2-data-0w82ls is "starting" 2022-03-08T23:13:53.724 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.722682 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:53.752 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.750400 I | op-osd: updating OSD 1 on PVC "smithi187-lv-2-data-0wdspq" 2022-03-08T23:13:53.752 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.750432 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:13:53.942 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.940670 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:53.960 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.958538 I | op-osd: updating OSD 10 on PVC "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:13:53.960 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:53.958564 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:13:54.038 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.036134 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:13:54.109 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.107369 I | op-mon: mons running: [b c a] 2022-03-08T23:13:54.123 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.121697 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:54.129 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.128031 I | op-osd: updating OSD 11 on PVC "smithi155-lv-4-data-04c4td" 2022-03-08T23:13:54.130 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.128060 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-4-data-04c4td" 2022-03-08T23:13:54.291 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.289085 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:54.330 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.328730 I | op-osd: updating OSD 2 on PVC "smithi187-lv-4-data-069kzv" 2022-03-08T23:13:54.330 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.328768 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:13:54.542 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.540465 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:13:54.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:54.730128 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:13:55.143 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:55.141349 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:13:55.143 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:55.141375 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:13:55.172 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:55.170791 I | op-mon: mons running: [b c a] 2022-03-08T23:13:55.437 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:55.435464 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:55.532 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:55.530769 I | op-osd: updating OSD 3 on PVC "smithi187-lv-3-data-0zsp69" 2022-03-08T23:13:55.533 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:55.530804 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:13:55.593 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:55.591583 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:13:55.930 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:55.928316 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:13:56.336 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:56.334025 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:13:56.336 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:56.334053 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:13:56.369 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:56.367508 I | op-mon: mons running: [b c a] 2022-03-08T23:13:56.636 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:56.634871 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:56.730 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:56.728569 I | op-osd: updating OSD 4 on PVC "smithi107-lv-2-data-09kfs9" 2022-03-08T23:13:56.730 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:56.728598 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:13:56.782 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:56.780271 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:13:56.782 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:56.780296 I | op-mon: mons created: 3 2022-03-08T23:13:57.203 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:57.201906 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:13:57.237 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:57.236290 I | op-mon: mons running: [b c a] 2022-03-08T23:13:57.453 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:57.451640 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:57.530 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:57.528898 I | op-osd: updating OSD 5 on PVC "smithi155-lv-3-data-08gx6j" 2022-03-08T23:13:57.531 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:57.528923 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:13:57.643 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:57.641397 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:13:57.643 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:57.641444 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:13:58.165 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:58.163599 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:13:58.233 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:58.231261 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:58.332 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:58.330514 I | op-osd: updating OSD 6 on PVC "smithi107-lv-4-data-0mptlv" 2022-03-08T23:13:58.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:58.330549 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:13:58.601 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:58.599993 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:13:59.030 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.028653 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:13:59.034 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.033018 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:59.142 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.139959 I | op-osd: updating OSD 7 on PVC "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:13:59.142 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.139983 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:13:59.550 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.548598 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:13:59.550 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.548621 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:13:59.836 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.834664 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:13:59.942 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.940640 I | op-osd: updating OSD 8 on PVC "smithi187-lv-1-data-067hpz" 2022-03-08T23:13:59.942 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.940678 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:13:59.967 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:13:59.965970 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:14:00.480 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.478525 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:14:00.481 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.478561 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:14:00.481 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.478569 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:14:00.636 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.634555 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:00.730 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.728987 I | op-osd: updating OSD 9 on PVC "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:14:00.731 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.729026 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:14:00.896 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.894015 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:14:00.942 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.940858 I | op-mgr: start running mgr 2022-03-08T23:14:00.942 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.940893 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:14:00.972 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:00.970693 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 9 on PVC "smithi107-lv-3-data-0r8x5p" LastHeartbeatTime:2022-03-08 23:14:00.936488598 +0000 UTC m=+531.474806997 LastTransitionTime:2022-03-08 23:14:00.936488518 +0000 UTC m=+531.474806927}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:14:01.555 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:01.553464 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:14:01.740 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:01.738631 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:14:01.893 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:01.891240 I | op-osd: OSD orchestration status for node smithi155-lv-2-data-0w82ls is "orchestrating" 2022-03-08T23:14:01.974 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:01.972440 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:14:02.059 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.057193 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:14:02.059 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.057467 I | op-mgr: successful modules: balancer 2022-03-08T23:14:02.071 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.069333 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:14:02.071 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.069379 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:14:02.150 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.147625 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:14:02.155 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152103 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-1" 2022-03-08T23:14:02.156 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152151 I | op-osd: OSD PVC "smithi107-lv-1-data-0mjtjg" already exists 2022-03-08T23:14:02.156 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152167 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-2" 2022-03-08T23:14:02.156 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152182 I | op-osd: OSD PVC "smithi107-lv-2-data-09kfs9" already exists 2022-03-08T23:14:02.156 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152203 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-3" 2022-03-08T23:14:02.156 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152218 I | op-osd: OSD PVC "smithi107-lv-3-data-0r8x5p" already exists 2022-03-08T23:14:02.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152232 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-4" 2022-03-08T23:14:02.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152248 I | op-osd: OSD PVC "smithi107-lv-4-data-0mptlv" already exists 2022-03-08T23:14:02.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152260 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-1" 2022-03-08T23:14:02.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152277 I | op-osd: OSD PVC "smithi155-lv-1-data-0c6ssv" already exists 2022-03-08T23:14:02.158 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152290 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-3" 2022-03-08T23:14:02.158 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152303 I | op-osd: OSD PVC "smithi155-lv-3-data-08gx6j" already exists 2022-03-08T23:14:02.158 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152315 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-4" 2022-03-08T23:14:02.158 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152329 I | op-osd: OSD PVC "smithi155-lv-4-data-04c4td" already exists 2022-03-08T23:14:02.158 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152340 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-1" 2022-03-08T23:14:02.159 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152354 I | op-osd: OSD PVC "smithi187-lv-1-data-067hpz" already exists 2022-03-08T23:14:02.159 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152366 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-2" 2022-03-08T23:14:02.159 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152379 I | op-osd: OSD PVC "smithi187-lv-2-data-0wdspq" already exists 2022-03-08T23:14:02.159 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152394 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-3" 2022-03-08T23:14:02.160 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152411 I | op-osd: OSD PVC "smithi187-lv-3-data-0zsp69" already exists 2022-03-08T23:14:02.160 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152427 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-4" 2022-03-08T23:14:02.160 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152441 I | op-osd: OSD PVC "smithi187-lv-4-data-069kzv" already exists 2022-03-08T23:14:02.160 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152453 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-2" 2022-03-08T23:14:02.160 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.152467 I | op-osd: OSD PVC "smithi155-lv-2-data-0w82ls" already exists 2022-03-08T23:14:02.293 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.291606 I | op-osd: OSD orchestration status for PVC smithi155-lv-2-data-0w82ls is "orchestrating" 2022-03-08T23:14:02.352 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348845 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:14:02.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348874 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-1-data-0mjtjg" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348882 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:14:02.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348887 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-2-data-09kfs9" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348892 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:14:02.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348897 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-3-data-0r8x5p" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348901 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:14:02.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348906 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-4-data-0mptlv" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348911 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:14:02.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348916 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-1-data-0c6ssv" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348921 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:14:02.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348926 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-3-data-08gx6j" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348937 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-4-data-04c4td" 2022-03-08T23:14:02.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348942 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-4-data-04c4td" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348946 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:14:02.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348951 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-1-data-067hpz" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348956 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:14:02.357 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348961 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-2-data-0wdspq" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.357 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348965 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:14:02.357 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348970 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-3-data-0zsp69" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.357 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348975 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:14:02.358 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348980 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-4-data-069kzv" because OSD daemon using the PVC already exists 2022-03-08T23:14:02.358 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.348985 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0w82ls" 2022-03-08T23:14:02.358 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.355319 I | op-osd: OSD orchestration status for PVC smithi155-lv-2-data-0w82ls is "starting" 2022-03-08T23:14:02.361 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.358854 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi155-lv-2-data-0w82ls. Status={Conditions:[] StartTime:2022-03-08 23:13:53 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-08T23:14:02.362 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.358873 I | op-osd: started OSD provisioning job for PVC "smithi155-lv-2-data-0w82ls" 2022-03-08T23:14:02.362 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.358880 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:14:02.363 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.358885 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:14:02.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.413535 I | op-osd: OSD orchestration status for PVC smithi155-lv-2-data-0w82ls is "starting" 2022-03-08T23:14:02.517 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.515660 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:02.569 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.567016 I | op-osd: updating OSD 1 on PVC "smithi187-lv-2-data-0wdspq" 2022-03-08T23:14:02.569 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.567044 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:14:02.792 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:02.790506 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:14:03.236 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:03.234830 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:03.331 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:03.329721 I | op-osd: updating OSD 10 on PVC "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:14:03.331 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:03.329758 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:14:03.872 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:03.870991 I | op-mgr: successful modules: prometheus 2022-03-08T23:14:03.910 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:03.909029 I | op-mgr: successful modules: dashboard 2022-03-08T23:14:04.034 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:04.032945 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:04.130 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:04.129117 I | op-osd: updating OSD 11 on PVC "smithi155-lv-4-data-04c4td" 2022-03-08T23:14:04.131 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:04.129144 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-4-data-04c4td" 2022-03-08T23:14:04.274 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:04.272537 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:14:04.836 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:04.833933 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:04.977 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:04.975263 I | op-osd: updating OSD 2 on PVC "smithi187-lv-4-data-069kzv" 2022-03-08T23:14:04.977 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:04.975364 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:14:05.634 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:05.632563 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:05.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:05.730058 I | op-osd: updating OSD 3 on PVC "smithi187-lv-3-data-0zsp69" 2022-03-08T23:14:05.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:05.730097 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:14:06.458 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:06.456725 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:06.532 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:06.530028 I | op-osd: updating OSD 4 on PVC "smithi107-lv-2-data-09kfs9" 2022-03-08T23:14:06.532 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:06.530058 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:14:07.235 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:07.233596 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:07.350 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:07.348067 I | op-osd: updating OSD 5 on PVC "smithi155-lv-3-data-08gx6j" 2022-03-08T23:14:07.350 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:07.348101 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:14:08.037 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:08.035966 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:08.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:08.130534 I | op-osd: updating OSD 6 on PVC "smithi107-lv-4-data-0mptlv" 2022-03-08T23:14:08.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:08.130568 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:14:08.435 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:08.433172 I | op-osd: OSD orchestration status for PVC smithi155-lv-2-data-0w82ls is "completed" 2022-03-08T23:14:08.435 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:08.433208 I | op-osd: creating OSD 0 on PVC "smithi155-lv-2-data-0w82ls" 2022-03-08T23:14:08.435 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:08.433223 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0w82ls" 2022-03-08T23:14:08.936 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:08.934351 I | op-osd: OSD orchestration status for PVC smithi155-lv-2-data-0w82ls is "completed" 2022-03-08T23:14:08.936 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:08.934375 I | op-osd: creating OSD 0 on PVC "smithi155-lv-2-data-0w82ls" 2022-03-08T23:14:08.937 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:08.934385 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0w82ls" 2022-03-08T23:14:09.629 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:09.627339 E | op-osd: failed to create OSD 0 on PVC "smithi155-lv-2-data-0w82ls": failed to create deployment for OSD 0 on PVC "smithi155-lv-2-data-0w82ls": deployments.apps "rook-ceph-osd-0" already exists 2022-03-08T23:14:09.735 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:09.733201 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:09.735 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:09.733319 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:14:09.736 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:09.733350 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:14:09.741 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:09.737510 I | op-osd: updating OSD 7 on PVC "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:14:09.741 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:09.737533 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:14:10.470 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:10.468739 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:10.529 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:10.527398 I | op-osd: updating OSD 8 on PVC "smithi187-lv-1-data-067hpz" 2022-03-08T23:14:10.529 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:10.527428 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:14:11.237 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:11.235791 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:11.330 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:11.328452 I | op-osd: updating OSD 9 on PVC "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:14:11.330 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:11.328473 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:14:11.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:11.946079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-08T23:14:11.948 INFO:tasks.rook.operator.smithi107.stdout:failed to create OSD 0 on PVC "smithi155-lv-2-data-0w82ls": failed to create deployment for OSD 0 on PVC "smithi155-lv-2-data-0w82ls": deployments.apps "rook-ceph-osd-0" already exists 2022-03-08T23:14:11.953 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:11.952146 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:14:12.020 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:12.018275 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:14:12.174 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:12.172325 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:14:16.183 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T23:14:16.184 INFO:teuthology.orchestra.run.smithi107.stdout:waiting for osd to come back up 2022-03-08T23:14:16.184 INFO:teuthology.orchestra.run.smithi107.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:14:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:++ toolbox ceph osd stat 2022-03-08T23:14:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:14:16.185 INFO:teuthology.orchestra.run.smithi107.stderr:++ cut -f3 -d ' ' 2022-03-08T23:14:16.254 INFO:teuthology.orchestra.run.smithi107.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:14:16.896 INFO:teuthology.orchestra.run.smithi107.stderr:+ num_osd=11 2022-03-08T23:14:16.896 INFO:teuthology.orchestra.run.smithi107.stderr:+ sleep 30s 2022-03-08T23:14:17.823 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:17.820937 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:14:17.823 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:17.820958 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T23:14:17.823 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:17.820971 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T23:14:17.836 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:17.834626 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:14:17.898 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:17.896423 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:14:17.902 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:17.896739 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:14:18.362 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:18.360571 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T23:14:18.481 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:18.479929 I | op-mon: start running mons 2022-03-08T23:14:18.529 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:18.528101 I | op-mon: parsing mon endpoints: b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 2022-03-08T23:14:18.554 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:18.552139 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.13.26:6789","10.97.212.176:6789","10.99.149.173:6789"]}] data:b=10.100.13.26:6789,c=10.97.212.176:6789,a=10.99.149.173:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:14:18.943 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:18.941489 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:14:18.943 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:18.941747 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:14:20.547 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:20.545190 I | op-mon: targeting the mon count 3 2022-03-08T23:14:20.554 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:20.552941 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:14:21.077 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:21.074269 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T23:14:21.078 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:21.074299 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:14:21.577 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:21.575469 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T23:14:21.578 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:21.575504 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:14:21.926 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:21.924782 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T23:14:21.927 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:21.924820 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:14:22.269 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:22.267955 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T23:14:22.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:22.267983 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T23:14:22.784 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:22.782097 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T23:14:22.784 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:22.782124 I | op-mon: checking for basic quorum with existing mons 2022-03-08T23:14:22.882 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:22.880970 I | op-mon: mon "b" endpoint is [v2:10.100.13.26:3300,v1:10.100.13.26:6789] 2022-03-08T23:14:22.985 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:22.983686 I | op-mon: mon "c" endpoint is [v2:10.97.212.176:3300,v1:10.97.212.176:6789] 2022-03-08T23:14:23.193 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:23.191874 I | op-mon: mon "a" endpoint is [v2:10.99.149.173:3300,v1:10.99.149.173:6789] 2022-03-08T23:14:23.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:23.794670 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.13.26:6789","10.97.212.176:6789","10.99.149.173:6789"]}] data:a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 mapping:{"node":{"a":{"Name":"smithi187","Hostname":"smithi187","Address":"172.21.15.187"},"b":{"Name":"smithi107","Hostname":"smithi107","Address":"172.21.15.107"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2] 2022-03-08T23:14:24.388 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:24.385872 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T23:14:24.388 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:24.386167 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T23:14:24.806 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:24.804157 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T23:14:24.820 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:24.818441 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T23:14:24.820 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:24.818502 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:14:25.395 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:25.393767 I | op-mon: mons running: [b c a] 2022-03-08T23:14:25.836 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:25.834150 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:14:25.841 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:25.839675 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T23:14:25.849 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:25.847930 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T23:14:25.849 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:25.847956 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:14:26.020 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:26.015610 I | op-mon: mons running: [b c a] 2022-03-08T23:14:26.443 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:26.441181 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:14:26.448 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:26.447318 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T23:14:26.457 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:26.455742 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T23:14:26.457 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:26.455765 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:14:26.596 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:26.594298 I | op-mon: mons running: [b c a] 2022-03-08T23:14:27.008 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:27.006858 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:14:27.009 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:27.006889 I | op-mon: mons created: 3 2022-03-08T23:14:27.420 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:27.418386 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T23:14:27.500 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:27.499145 I | op-mon: mons running: [b c a] 2022-03-08T23:14:27.897 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:27.895732 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T23:14:27.898 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:27.895780 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T23:14:28.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:28.351291 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T23:14:28.911 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:28.909802 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T23:14:29.322 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:29.320407 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T23:14:29.758 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:29.756736 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T23:14:29.759 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:29.756761 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T23:14:30.290 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:30.288386 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T23:14:30.690 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:30.687972 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T23:14:30.690 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:30.688017 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T23:14:30.690 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:30.688035 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T23:14:31.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.110855 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T23:14:31.141 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.139734 I | op-mgr: start running mgr 2022-03-08T23:14:31.141 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.139769 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T23:14:31.589 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.588049 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T23:14:31.597 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.596158 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T23:14:31.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.599217 I | op-mgr: setting services to point to mgr "a" 2022-03-08T23:14:31.741 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.739214 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T23:14:31.741 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.739362 I | op-mgr: successful modules: balancer 2022-03-08T23:14:31.755 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.754016 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T23:14:31.756 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.754047 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T23:14:31.812 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.810221 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T23:14:31.943 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940742 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-1" 2022-03-08T23:14:31.944 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940781 I | op-osd: OSD PVC "smithi107-lv-1-data-0mjtjg" already exists 2022-03-08T23:14:31.944 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940792 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-2" 2022-03-08T23:14:31.944 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940807 I | op-osd: OSD PVC "smithi107-lv-2-data-09kfs9" already exists 2022-03-08T23:14:31.944 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940814 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-3" 2022-03-08T23:14:31.945 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940823 I | op-osd: OSD PVC "smithi107-lv-3-data-0r8x5p" already exists 2022-03-08T23:14:31.945 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940831 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi107-lv-4" 2022-03-08T23:14:31.945 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940840 I | op-osd: OSD PVC "smithi107-lv-4-data-0mptlv" already exists 2022-03-08T23:14:31.945 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940847 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-1" 2022-03-08T23:14:31.945 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940856 I | op-osd: OSD PVC "smithi155-lv-1-data-0c6ssv" already exists 2022-03-08T23:14:31.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940866 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-3" 2022-03-08T23:14:31.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940875 I | op-osd: OSD PVC "smithi155-lv-3-data-08gx6j" already exists 2022-03-08T23:14:31.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940882 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-4" 2022-03-08T23:14:31.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940893 I | op-osd: OSD PVC "smithi155-lv-4-data-04c4td" already exists 2022-03-08T23:14:31.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940900 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-1" 2022-03-08T23:14:31.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940909 I | op-osd: OSD PVC "smithi187-lv-1-data-067hpz" already exists 2022-03-08T23:14:31.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940915 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-2" 2022-03-08T23:14:31.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940924 I | op-osd: OSD PVC "smithi187-lv-2-data-0wdspq" already exists 2022-03-08T23:14:31.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940932 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-3" 2022-03-08T23:14:31.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940942 I | op-osd: OSD PVC "smithi187-lv-3-data-0zsp69" already exists 2022-03-08T23:14:31.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940948 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi187-lv-4" 2022-03-08T23:14:31.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940957 I | op-osd: OSD PVC "smithi187-lv-4-data-069kzv" already exists 2022-03-08T23:14:31.949 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940963 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi155-lv-2" 2022-03-08T23:14:31.949 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:31.940971 I | op-osd: OSD PVC "smithi155-lv-2-data-0w82ls" already exists 2022-03-08T23:14:32.003 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000280 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:14:32.003 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000299 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-1-data-0mjtjg" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.003 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000302 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:14:32.003 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000305 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-2-data-09kfs9" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000308 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:14:32.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000310 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-3-data-0r8x5p" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000313 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:14:32.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000315 I | op-osd: skipping OSD prepare job creation for PVC "smithi107-lv-4-data-0mptlv" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000318 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:14:32.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000320 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-1-data-0c6ssv" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000323 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:14:32.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000325 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-3-data-08gx6j" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000328 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-4-data-04c4td" 2022-03-08T23:14:32.006 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000330 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-4-data-04c4td" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.006 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000333 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:14:32.006 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000335 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-1-data-067hpz" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.006 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000339 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:14:32.007 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000341 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-2-data-0wdspq" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.007 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000344 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:14:32.008 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000346 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-3-data-0zsp69" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.008 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000349 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:14:32.008 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000353 I | op-osd: skipping OSD prepare job creation for PVC "smithi187-lv-4-data-069kzv" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.008 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000356 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0w82ls" 2022-03-08T23:14:32.009 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000358 I | op-osd: skipping OSD prepare job creation for PVC "smithi155-lv-2-data-0w82ls" because OSD daemon using the PVC already exists 2022-03-08T23:14:32.009 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000361 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T23:14:32.009 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.000364 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T23:14:32.241 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.239967 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:32.246 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.245057 I | op-osd: updating OSD 0 on PVC "smithi155-lv-2-data-0w82ls" 2022-03-08T23:14:32.246 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.245076 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-2-data-0w82ls" 2022-03-08T23:14:32.419 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.416925 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:32.427 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.425636 I | op-osd: updating OSD 1 on PVC "smithi187-lv-2-data-0wdspq" 2022-03-08T23:14:32.427 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.425770 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-2-data-0wdspq" 2022-03-08T23:14:32.736 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.734568 I | op-mgr: successful modules: prometheus 2022-03-08T23:14:32.755 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.752930 I | op-mgr: successful modules: dashboard 2022-03-08T23:14:32.774 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.772556 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:32.778 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.776808 I | op-osd: updating OSD 10 on PVC "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:14:32.778 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.776829 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-1-data-0mjtjg" 2022-03-08T23:14:32.798 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:32.796892 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T23:14:33.076 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:33.074532 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:33.170 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:33.168552 I | op-osd: updating OSD 11 on PVC "smithi155-lv-4-data-04c4td" 2022-03-08T23:14:33.170 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:33.168589 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-4-data-04c4td" 2022-03-08T23:14:33.875 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:33.873335 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:33.969 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:33.967248 I | op-osd: updating OSD 2 on PVC "smithi187-lv-4-data-069kzv" 2022-03-08T23:14:33.969 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:33.967275 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-4-data-069kzv" 2022-03-08T23:14:34.203 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:34.201285 I | op-mgr: successful modules: orchestrator modules 2022-03-08T23:14:34.678 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:34.676116 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:34.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:34.769556 I | op-osd: updating OSD 3 on PVC "smithi187-lv-3-data-0zsp69" 2022-03-08T23:14:34.772 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:34.769590 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-3-data-0zsp69" 2022-03-08T23:14:35.475 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:35.472854 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:35.584 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:35.582664 I | op-osd: updating OSD 4 on PVC "smithi107-lv-2-data-09kfs9" 2022-03-08T23:14:35.584 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:35.582689 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-2-data-09kfs9" 2022-03-08T23:14:36.277 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:36.275538 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:36.371 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:36.369285 I | op-osd: updating OSD 5 on PVC "smithi155-lv-3-data-08gx6j" 2022-03-08T23:14:36.371 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:36.369322 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-3-data-08gx6j" 2022-03-08T23:14:37.077 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:37.075696 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:37.204 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:37.202779 I | op-osd: updating OSD 6 on PVC "smithi107-lv-4-data-0mptlv" 2022-03-08T23:14:37.205 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:37.202814 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-4-data-0mptlv" 2022-03-08T23:14:37.876 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:37.874627 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:37.972 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:37.970009 I | op-osd: updating OSD 7 on PVC "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:14:37.972 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:37.970046 I | op-osd: OSD will have its main bluestore block on "smithi155-lv-1-data-0c6ssv" 2022-03-08T23:14:38.673 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:38.671087 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:38.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:38.769265 I | op-osd: updating OSD 8 on PVC "smithi187-lv-1-data-067hpz" 2022-03-08T23:14:38.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:38.769300 I | op-osd: OSD will have its main bluestore block on "smithi187-lv-1-data-067hpz" 2022-03-08T23:14:39.478 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:39.476606 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T23:14:39.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:39.575381 I | op-osd: updating OSD 9 on PVC "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:14:39.584 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:39.582654 I | op-osd: OSD will have its main bluestore block on "smithi107-lv-3-data-0r8x5p" 2022-03-08T23:14:40.607 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:40.605105 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T23:14:40.607 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:14:40.605124 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T23:14:46.897 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T23:14:46.898 INFO:teuthology.orchestra.run.smithi107.stdout:waiting for osd to come back up 2022-03-08T23:14:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T23:14:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:++ toolbox ceph osd stat 2022-03-08T23:14:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T23:14:46.899 INFO:teuthology.orchestra.run.smithi107.stderr:++ cut -f3 -d ' ' 2022-03-08T23:14:46.966 INFO:teuthology.orchestra.run.smithi107.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T23:14:47.433 INFO:teuthology.orchestra.run.smithi107.stderr:+ num_osd=12 2022-03-08T23:14:47.434 INFO:teuthology.orchestra.run.smithi107.stderr:+ sleep 30s 2022-03-08T23:15:17.436 INFO:teuthology.orchestra.run.smithi107.stderr:+ '[' 12 '!=' 12 ']' 2022-03-08T23:15:17.437 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T23:15:17.453 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch status 2022-03-08T23:15:20.906 INFO:teuthology.orchestra.run.smithi107.stdout:Backend: rook 2022-03-08T23:15:20.906 INFO:teuthology.orchestra.run.smithi107.stdout:Available: Yes 2022-03-08T23:15:20.933 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ps 2022-03-08T23:15:21.474 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T23:15:21.474 INFO:teuthology.orchestra.run.smithi107.stdout:crashcollector.crash smithi107 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T23:15:21.475 INFO:teuthology.orchestra.run.smithi107.stdout:crashcollector.crash smithi155 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.475 INFO:teuthology.orchestra.run.smithi107.stdout:crashcollector.crash smithi187 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T23:15:21.475 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.a smithi155 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T23:15:21.475 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi187 running (8m) 0s ago 8m - - 4b871cde5e1a 2022-03-08T23:15:21.476 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi107 running (8m) 0s ago 8m - - 4b871cde5e1a 2022-03-08T23:15:21.476 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi155 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T23:15:21.476 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi155 running (72s) 0s ago 72s - - 4b871cde5e1a 2022-03-08T23:15:21.476 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi187 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.477 INFO:teuthology.orchestra.run.smithi107.stdout:osd.10 smithi107 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.477 INFO:teuthology.orchestra.run.smithi107.stdout:osd.11 smithi155 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T23:15:21.477 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi187 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.477 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi187 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.477 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi107 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.478 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi155 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.478 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi107 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.478 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi155 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.478 INFO:teuthology.orchestra.run.smithi107.stdout:osd.8 smithi187 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.479 INFO:teuthology.orchestra.run.smithi107.stdout:osd.9 smithi107 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T23:15:21.500 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls 2022-03-08T23:15:22.185 INFO:teuthology.orchestra.run.smithi107.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T23:15:22.186 INFO:teuthology.orchestra.run.smithi107.stdout:crash 3/3 0s ago 5m * 2022-03-08T23:15:22.186 INFO:teuthology.orchestra.run.smithi107.stdout:mgr 1/1 0s ago 5m count:1 2022-03-08T23:15:22.186 INFO:teuthology.orchestra.run.smithi107.stdout:mon 3/3 0s ago 8m count:3 2022-03-08T23:15:22.186 INFO:teuthology.orchestra.run.smithi107.stdout:osd 24 0s ago 4m 2022-03-08T23:15:22.187 INFO:teuthology.orchestra.run.smithi107.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T23:15:22.212 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch host ls 2022-03-08T23:15:22.789 INFO:teuthology.orchestra.run.smithi107.stdout:HOST ADDR LABELS STATUS 2022-03-08T23:15:22.789 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 172.21.15.107/smithi107 2022-03-08T23:15:22.789 INFO:teuthology.orchestra.run.smithi107.stdout:smithi155 172.21.15.155/smithi155 2022-03-08T23:15:22.789 INFO:teuthology.orchestra.run.smithi107.stdout:smithi187 172.21.15.187/smithi187 2022-03-08T23:15:22.790 INFO:teuthology.orchestra.run.smithi107.stdout:3 hosts in cluster 2022-03-08T23:15:22.810 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch device ls 2022-03-08T23:15:23.324 INFO:teuthology.orchestra.run.smithi107.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T23:15:23.324 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:15:23.324 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:15:23.325 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:15:23.325 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:15:23.325 INFO:teuthology.orchestra.run.smithi107.stdout:smithi155 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:15:23.325 INFO:teuthology.orchestra.run.smithi107.stdout:smithi155 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:15:23.326 INFO:teuthology.orchestra.run.smithi107.stdout:smithi155 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:15:23.326 INFO:teuthology.orchestra.run.smithi107.stdout:smithi155 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:15:23.326 INFO:teuthology.orchestra.run.smithi107.stdout:smithi187 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T23:15:23.326 INFO:teuthology.orchestra.run.smithi107.stdout:smithi187 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T23:15:23.326 INFO:teuthology.orchestra.run.smithi107.stdout:smithi187 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T23:15:23.327 INFO:teuthology.orchestra.run.smithi107.stdout:smithi187 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T23:15:23.346 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch apply rgw foo 2022-03-08T23:15:23.900 INFO:teuthology.orchestra.run.smithi107.stdout:Created 2022-03-08T23:15:23.902 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:23.899857 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-08T23:15:23.921 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch apply mds foo 2022-03-08T23:15:24.016 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:24.014773 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:24.017 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:24.014856 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:15:24.492 INFO:teuthology.orchestra.run.smithi107.stdout:Created 2022-03-08T23:15:24.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:24.492778 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:15:24.516 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch apply rbd-mirror 2022-03-08T23:15:24.548 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:24.546140 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:24.548 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:24.546236 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:15:25.241 INFO:teuthology.orchestra.run.smithi107.stdout:Success 2022-03-08T23:15:25.263 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch apply nfs foo --port 12777 2022-03-08T23:15:25.299 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:25.297146 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:25.299 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:25.297236 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T23:15:29.200 INFO:teuthology.orchestra.run.smithi107.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-08T23:15:29.201 INFO:teuthology.orchestra.run.smithi107.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-08T23:15:29.224 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-08T23:15:29.240 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:15:29.952 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:15:29.953 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:28.000000Z", "last_refresh": "2022-03-08T23:15:29.744274Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:29.744274Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:15:29.744274Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:15:29.744274Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:15:29.744274Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:29.744274Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-08T23:15:29.744274Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:29.744274Z", "running": 0, "size": 1}}] 2022-03-08T23:15:29.977 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch rm mds.foo 2022-03-08T23:15:30.531 INFO:teuthology.orchestra.run.smithi107.stdout:Removed cephfilesystems/foo 2022-03-08T23:15:30.550 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch rm rbd-mirror 2022-03-08T23:15:31.103 INFO:teuthology.orchestra.run.smithi107.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-08T23:15:31.126 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch rm rgw.foo 2022-03-08T23:15:32.185 INFO:teuthology.orchestra.run.smithi107.stdout:Removed cephobjectstores/foo 2022-03-08T23:15:32.206 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:15:32.968 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:15:32.969 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:28.000000Z", "last_refresh": "2022-03-08T23:15:32.740235Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:32.740235Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:15:32.740235Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:15:32.740235Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:15:32.740235Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:32.740235Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:32.740235Z", "running": 0, "size": 1}}] 2022-03-08T23:15:33.870 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:33.868259 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:15:35.574 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:35.572868 I | ceph-object-controller: reconciling object store deployments 2022-03-08T23:15:35.719 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:35.718016 I | ceph-object-controller: ceph object store gateway service running at 10.102.249.102 2022-03-08T23:15:35.720 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:35.718045 I | ceph-object-controller: reconciling object store pools 2022-03-08T23:15:36.042 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:36.040644 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T23:15:36.605 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:36.603196 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T23:15:37.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:37.945343 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-08T23:15:38.580 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:38.575011 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-08T23:15:40.238 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:40.236483 I | op-mds: setting mds config flags 2022-03-08T23:15:40.238 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:40.236516 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:15:41.841 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:41.839656 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:15:41.851 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:41.849733 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-08T23:15:42.147 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:42.143326 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi187": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:15:42.997 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:15:43.087 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:43.080759 I | op-mds: setting mds config flags 2022-03-08T23:15:43.087 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:43.080798 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:15:43.490 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:43.488824 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-08T23:15:43.496 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:43.494320 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-08T23:15:43.499 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:43.494587 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-08T23:15:43.500 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:43.496224 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-08T23:15:43.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:43.649464 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T23:15:44.279 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:15:44.280 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:28.000000Z", "last_refresh": "2022-03-08T23:15:43.806699Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:43.806699Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:15:43.806699Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:15:43.806699Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:15:43.806699Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:43.806699Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:43.806699Z", "running": 0, "size": 1}}] 2022-03-08T23:15:44.642 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:44.571820 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-08T23:15:46.941 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:46.939262 I | ceph-file-controller: creating filesystem "foo" 2022-03-08T23:15:47.044 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:47.042261 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-08T23:15:47.117 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:47.115133 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:51.868 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:51.866558 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:15:51.856+0000 7f57ebb6f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:15:51.869 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:51.856+0000 7f57ebb6f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:15:51.869 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:51.856+0000 7f57ebb6f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:15:51.869 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:15:52.344 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:52.342013 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:53.192 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:53.190807 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-08T23:15:53.335 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:53.333711 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:15:53.324+0000 7fc1622fd380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:15:53.336 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:53.324+0000 7fc1622fd380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:15:53.336 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:53.324+0000 7fc1622fd380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:15:53.336 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:15:53.514 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:53.512754 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:54.305 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:15:55.325 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:55.323723 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:15:55.316+0000 7f71e3dd3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:15:55.326 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:55.316+0000 7f71e3dd3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:15:55.326 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:55.316+0000 7f71e3dd3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:15:55.326 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:15:56.169 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:56.167179 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:56.572 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:15:56.572 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:28.000000Z", "last_refresh": "2022-03-08T23:15:55.486753Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:55.486753Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:15:55.486753Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:15:55.486753Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:15:55.486753Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:15:55.486753Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:15:55.486753Z", "running": 0, "size": 1}}] 2022-03-08T23:15:56.992 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:56.990674 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:15:56.980+0000 7fa52af09380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:15:56.993 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:56.980+0000 7fa52af09380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:15:56.993 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:56.980+0000 7fa52af09380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:15:56.993 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:15:57.008 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:57.006875 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:57.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:57.598021 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:15:57.588+0000 7f096fa3d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:15:57.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:57.588+0000 7f096fa3d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:15:57.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:57.588+0000 7f096fa3d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:15:57.601 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:15:57.616 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:57.615234 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:58.718 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:58.715913 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:15:58.708+0000 7f221dffe380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:15:58.718 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:58.708+0000 7f221dffe380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:15:58.718 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:58.708+0000 7f221dffe380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:15:58.719 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:15:58.767 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:58.765548 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:15:59.556 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:59.554898 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-08T23:15:59.939 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:59.937236 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:15:59.928+0000 7fe6fb473380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:15:59.939 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:59.928+0000 7fe6fb473380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:15:59.940 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:15:59.928+0000 7fe6fb473380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:15:59.940 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:15:59.950 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:15:59.948374 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:16:00.275 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:00.273384 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-08T23:16:00.514 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:00.512591 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:16:00.504+0000 7f5617300380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:16:00.515 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:00.504+0000 7f5617300380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:16:00.515 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:00.504+0000 7f5617300380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:16:00.515 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:16:00.589 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:00.587967 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:16:01.498 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:01.495719 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:16:01.488+0000 7f7bc16dd380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:16:01.499 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:01.488+0000 7f7bc16dd380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:16:01.499 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:01.488+0000 7f7bc16dd380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:16:01.499 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:16:02.242 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:02.239501 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-08T23:16:02.242 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:02.239526 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-08T23:16:04.081 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:04.078199 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:16:04.082 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:04.078271 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:16:04.085 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:04.080128 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-08T23:16:04.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:04.730126 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:16:04.720+0000 7f57d2db6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:16:04.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:04.720+0000 7f57d2db6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:16:04.733 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:04.720+0000 7f57d2db6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:16:04.733 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:16:04.923 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:04.921746 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:16:05.561 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:05.559235 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:16:05.561 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:05.559261 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-08T23:16:05.562 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:05.559270 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-08T23:16:06.081 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:06.079268 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-08T23:16:06.594 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:16:06.715 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:06.713922 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T23:16:07.254 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:07.251969 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T23:16:07.254 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:07.251998 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-08T23:16:07.254 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:07.252005 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-08T23:16:07.351 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:16:07.352 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:16:07.163508Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:07.163508Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:16:07.163508Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:16:07.163508Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:16:07.163508Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:07.163508Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:07.163508Z", "running": 0, "size": 1}}] 2022-03-08T23:16:08.040 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:08.038723 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-08T23:16:08.041 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:08.038753 I | ceph-file-controller: downing filesystem "foo" 2022-03-08T23:16:09.457 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:09.455573 I | ceph-file-controller: downed filesystem "foo" 2022-03-08T23:16:09.904 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:09.902918 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:16:12.096 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:12.094944 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:16:12.088+0000 7f70e838d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:16:12.097 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:12.088+0000 7f70e838d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:16:12.097 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:12.088+0000 7f70e838d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:16:12.097 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:16:14.731 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:14.729365 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-08T23:16:14.734 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:14.729444 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-08T23:16:17.373 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:16:17.620 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:17.618580 I | cephclient: purge completed for pool "foo-metadata" 2022-03-08T23:16:18.065 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:18.063150 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-08T23:16:18.065 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:18.063169 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-08T23:16:18.074 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:16:18.074 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:16:17.874361Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:17.874361Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:16:17.874361Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:16:17.874361Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:16:17.874361Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:17.874361Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:17.874361Z", "running": 0, "size": 1}}] 2022-03-08T23:16:20.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:20.106593 I | cephclient: purge completed for pool "foo-data0" 2022-03-08T23:16:20.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:20.106670 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T23:16:20.151 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:20.148303 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-08T23:16:20.189 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:20.171569 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-08T23:16:20.194 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:20.172160 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-08T23:16:20.194 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:20.172438 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-08T23:16:22.344 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:22.343067 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:16:23.352 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:23.350437 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:16:23.340+0000 7f3e346ec380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:16:23.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:23.340+0000 7f3e346ec380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:16:23.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:23.340+0000 7f3e346ec380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:16:23.353 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:16:23.570 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:23.568691 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi187": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:16:24.153 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:24.151691 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi107": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T23:16:28.095 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:16:29.474 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:16:29.474 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:16:29.068909Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:16:29.068909Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:16:29.068909Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:16:29.068909Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:29.068909Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:29.068909Z", "running": 0, "size": 1}}] 2022-03-08T23:16:39.496 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:16:40.385 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:16:40.385 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:16:40.220653Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:16:40.220653Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:16:40.220653Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:16:40.220653Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:40.220653Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:40.220653Z", "running": 0, "size": 1}}] 2022-03-08T23:16:43.838 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:43.836777 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:16:44.520 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:16:44.518780 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:16:44.507+0000 7f3a42d66380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:16:44.521 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:44.507+0000 7f3a42d66380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:16:44.521 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:16:44.507+0000 7f3a42d66380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:16:44.522 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:16:50.411 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:16:51.102 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:16:51.103 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:16:50.937979Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:16:50.937979Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:16:50.937979Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:16:50.937979Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:16:50.937979Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:16:50.937979Z", "running": 0, "size": 1}}] 2022-03-08T23:17:01.131 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:17:01.889 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:17:01.889 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:17:01.718826Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:17:01.718826Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:17:01.718826Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:17:01.718826Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:01.718826Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:01.718826Z", "running": 0, "size": 1}}] 2022-03-08T23:17:11.920 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:17:12.633 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:17:12.634 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:17:12.436586Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:17:12.436586Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:17:12.436586Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:17:12.436586Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:12.436586Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:12.436586Z", "running": 0, "size": 1}}] 2022-03-08T23:17:22.661 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:17:23.338 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:17:23.338 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:17:23.151750Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:17:23.151750Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:17:23.151750Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:17:23.151750Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:23.151750Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:23.151750Z", "running": 0, "size": 1}}] 2022-03-08T23:17:25.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:17:25.490527 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:17:26.127 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:17:26.125179 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:17:26.115+0000 7f1070e2b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:17:26.127 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:17:26.115+0000 7f1070e2b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:17:26.128 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:17:26.115+0000 7f1070e2b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:17:26.128 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:17:33.364 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:17:34.062 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:17:34.062 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:17:33.864876Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:17:33.864876Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:17:33.864876Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:17:33.864876Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:33.864876Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:33.864876Z", "running": 0, "size": 1}}] 2022-03-08T23:17:44.089 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:17:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:17:44.823 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:17:44.633926Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:17:44.633926Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:17:44.633926Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:17:44.633926Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:44.633926Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:44.633926Z", "running": 0, "size": 1}}] 2022-03-08T23:17:54.849 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:17:55.607 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:17:55.607 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:17:55.444591Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:17:55.444591Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:17:55.444591Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:17:55.444591Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:17:55.444591Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:17:55.444591Z", "running": 0, "size": 1}}] 2022-03-08T23:18:05.629 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:18:06.321 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:18:06.322 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:18:06.138506Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:18:06.138506Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:18:06.138506Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:18:06.138506Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:06.138506Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:06.138506Z", "running": 0, "size": 1}}] 2022-03-08T23:18:16.346 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:18:17.010 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:18:17.011 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:18:16.840540Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:18:16.840540Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:18:16.840540Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:18:16.840540Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:16.840540Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:16.840540Z", "running": 0, "size": 1}}] 2022-03-08T23:18:27.036 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:18:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:18:27.711 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:18:27.537535Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:18:27.537535Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:18:27.537535Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:18:27.537535Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:27.537535Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:27.537535Z", "running": 0, "size": 1}}] 2022-03-08T23:18:37.734 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:18:38.445 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:18:38.446 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:18:38.270339Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:18:38.270339Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:18:38.270339Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:18:38.270339Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:38.270339Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:38.270339Z", "running": 0, "size": 1}}] 2022-03-08T23:18:48.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:18:48.052143 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:18:48.470 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:18:48.698 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:18:48.696177 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:18:48.689+0000 7f58a8415380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:18:48.698 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:18:48.689+0000 7f58a8415380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:18:48.698 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:18:48.689+0000 7f58a8415380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:18:48.699 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:18:49.191 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:18:49.191 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:18:48.993187Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:18:48.993187Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:18:48.993187Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:18:48.993187Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:48.993187Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:48.993187Z", "running": 0, "size": 1}}] 2022-03-08T23:18:59.217 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:19:00.018 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:19:00.019 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:18:59.837899Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:18:59.837899Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:18:59.837899Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:18:59.837899Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:18:59.837899Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:18:59.837899Z", "running": 0, "size": 1}}] 2022-03-08T23:19:10.043 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:19:10.724 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:19:10.725 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:19:10.554916Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:19:10.554916Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:19:10.554916Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:19:10.554916Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:10.554916Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:10.554916Z", "running": 0, "size": 1}}] 2022-03-08T23:19:20.753 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:19:21.441 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:19:21.441 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:19:21.252222Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:19:21.252222Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:19:21.252222Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:19:21.252222Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:21.252222Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:21.252222Z", "running": 0, "size": 1}}] 2022-03-08T23:19:31.469 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:19:32.193 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:19:32.193 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:19:31.975447Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:19:31.975447Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:19:31.975447Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:19:31.975447Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:31.975447Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:31.975447Z", "running": 0, "size": 1}}] 2022-03-08T23:19:42.220 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:19:42.904 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:19:42.904 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:19:42.737709Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:19:42.737709Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:19:42.737709Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:19:42.737709Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:42.737709Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:42.737709Z", "running": 0, "size": 1}}] 2022-03-08T23:19:52.927 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:19:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:19:53.609 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:19:53.414194Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:19:53.414194Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:19:53.414194Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:19:53.414194Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:19:53.414194Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:19:53.414194Z", "running": 0, "size": 1}}] 2022-03-08T23:20:03.634 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:20:04.373 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:20:04.373 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:20:04.170214Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:20:04.170214Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:20:04.170214Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:20:04.170214Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:04.170214Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:04.170214Z", "running": 0, "size": 1}}] 2022-03-08T23:20:14.399 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:20:15.070 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:20:15.070 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:20:14.889948Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:20:14.889948Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:20:14.889948Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:20:14.889948Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:14.889948Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:14.889948Z", "running": 0, "size": 1}}] 2022-03-08T23:20:25.096 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:20:25.807 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:20:25.808 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:20:25.633985Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:20:25.633985Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:20:25.633985Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:20:25.633985Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:25.633985Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:25.633985Z", "running": 0, "size": 1}}] 2022-03-08T23:20:35.834 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:20:36.584 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:20:36.584 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:20:36.378019Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:20:36.378019Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:20:36.378019Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:20:36.378019Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:36.378019Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:36.378019Z", "running": 0, "size": 1}}] 2022-03-08T23:20:46.610 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:20:47.285 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:20:47.286 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:20:47.109977Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:20:47.109977Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:20:47.109977Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:20:47.109977Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:47.109977Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:47.109977Z", "running": 0, "size": 1}}] 2022-03-08T23:20:57.318 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:20:58.044 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:20:58.045 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:20:57.817510Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:20:57.817510Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:20:57.817510Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:20:57.817510Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:20:57.817510Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:20:57.817510Z", "running": 0, "size": 1}}] 2022-03-08T23:21:08.067 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:21:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:21:08.739 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:21:08.571609Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:21:08.571609Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:21:08.571609Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:21:08.571609Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:08.571609Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:08.571609Z", "running": 0, "size": 1}}] 2022-03-08T23:21:18.766 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:21:19.485 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:21:19.486 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:21:19.250151Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:21:19.250151Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:21:19.250151Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:21:19.250151Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:19.250151Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:19.250151Z", "running": 0, "size": 1}}] 2022-03-08T23:21:29.511 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:21:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:21:30.235 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:21:30.019077Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:21:30.019077Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:21:30.019077Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:21:30.019077Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:30.019077Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:30.019077Z", "running": 0, "size": 1}}] 2022-03-08T23:21:32.548 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:21:32.546711 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:21:33.197 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:21:33.195929 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:21:33.183+0000 7f529df62380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:21:33.198 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:21:33.183+0000 7f529df62380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:21:33.198 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:21:33.183+0000 7f529df62380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:21:33.198 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:21:40.259 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:21:40.973 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:21:40.973 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:21:40.787486Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:21:40.787486Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:21:40.787486Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:21:40.787486Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:40.787486Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:40.787486Z", "running": 0, "size": 1}}] 2022-03-08T23:21:50.998 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:21:51.746 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:21:51.746 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:21:51.559368Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:21:51.559368Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:21:51.559368Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:21:51.559368Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:21:51.559368Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:21:51.559368Z", "running": 0, "size": 1}}] 2022-03-08T23:22:01.768 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:22:02.508 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:22:02.509 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:22:02.316485Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:22:02.316485Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:22:02.316485Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:22:02.316485Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:02.316485Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:02.316485Z", "running": 0, "size": 1}}] 2022-03-08T23:22:12.531 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:22:13.247 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:22:13.247 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:22:13.057696Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:22:13.057696Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:22:13.057696Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:22:13.057696Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:13.057696Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:13.057696Z", "running": 0, "size": 1}}] 2022-03-08T23:22:23.272 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:22:23.943 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:22:23.944 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:22:23.784086Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:22:23.784086Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:22:23.784086Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:22:23.784086Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:23.784086Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:23.784086Z", "running": 0, "size": 1}}] 2022-03-08T23:22:33.967 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:22:34.725 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:22:34.725 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:22:34.480290Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:22:34.480290Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:22:34.480290Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:22:34.480290Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:34.480290Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:34.480290Z", "running": 0, "size": 1}}] 2022-03-08T23:22:44.752 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:22:45.490 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:22:45.490 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:22:45.258232Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:22:45.258232Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:22:45.258232Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:22:45.258232Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:45.258232Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:45.258232Z", "running": 0, "size": 1}}] 2022-03-08T23:22:55.514 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:22:56.299 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:22:56.299 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:22:56.098514Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:22:56.098514Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:22:56.098514Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:22:56.098514Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:22:56.098514Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:22:56.098514Z", "running": 0, "size": 1}}] 2022-03-08T23:23:06.326 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:23:07.287 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:23:07.287 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:23:06.822102Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:23:06.822102Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:23:06.822102Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:23:06.822102Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:06.822102Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:06.822102Z", "running": 0, "size": 1}}] 2022-03-08T23:23:17.323 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:23:18.113 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:23:18.113 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:23:17.897342Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:23:17.897342Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:23:17.897342Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:23:17.897342Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:17.897342Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:17.897342Z", "running": 0, "size": 1}}] 2022-03-08T23:23:28.137 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:23:28.824 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:23:28.824 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:23:28.643124Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:23:28.643124Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:23:28.643124Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:23:28.643124Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:28.643124Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:28.643124Z", "running": 0, "size": 1}}] 2022-03-08T23:23:38.850 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:23:39.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:23:39.540 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:23:39.355231Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:23:39.355231Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:23:39.355231Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:23:39.355231Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:39.355231Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:39.355231Z", "running": 0, "size": 1}}] 2022-03-08T23:23:49.564 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:23:50.288 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:23:50.289 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:23:50.082204Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:23:50.082204Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:23:50.082204Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:23:50.082204Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:23:50.082204Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:23:50.082204Z", "running": 0, "size": 1}}] 2022-03-08T23:24:00.315 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:24:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:24:01.025 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:24:00.795706Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:24:00.795706Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:24:00.795706Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:24:00.795706Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:00.795706Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:00.795706Z", "running": 0, "size": 1}}] 2022-03-08T23:24:11.048 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:24:11.731 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:24:11.731 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:24:11.554456Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:24:11.554456Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:24:11.554456Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:24:11.554456Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:11.554456Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:11.554456Z", "running": 0, "size": 1}}] 2022-03-08T23:24:21.755 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:24:22.509 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:24:22.510 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:24:22.249095Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:24:22.249095Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:24:22.249095Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:24:22.249095Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:22.249095Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:22.249095Z", "running": 0, "size": 1}}] 2022-03-08T23:24:32.534 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:24:33.275 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:24:33.275 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:24:33.093034Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:24:33.093034Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:24:33.093034Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:24:33.093034Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:33.093034Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:33.093034Z", "running": 0, "size": 1}}] 2022-03-08T23:24:43.301 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:24:43.981 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:24:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:24:43.777109Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:24:43.777109Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:24:43.777109Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:24:43.777109Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:43.777109Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:43.777109Z", "running": 0, "size": 1}}] 2022-03-08T23:24:54.005 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:24:54.793 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:24:54.794 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:24:54.543133Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:24:54.543133Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:24:54.543133Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:24:54.543133Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:24:54.543133Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:24:54.543133Z", "running": 0, "size": 1}}] 2022-03-08T23:25:04.819 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:25:05.613 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:25:05.613 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:25:05.318250Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:25:05.318250Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:25:05.318250Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:25:05.318250Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:05.318250Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:05.318250Z", "running": 0, "size": 1}}] 2022-03-08T23:25:15.635 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:25:16.319 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:25:16.319 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:25:16.145091Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:25:16.145091Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:25:16.145091Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:25:16.145091Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:16.145091Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:16.145091Z", "running": 0, "size": 1}}] 2022-03-08T23:25:26.347 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:25:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:25:29.838 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:25:29.646959Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:25:29.646959Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:25:29.646959Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:25:29.646959Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:29.646959Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:29.646959Z", "running": 0, "size": 1}}] 2022-03-08T23:25:39.862 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:25:40.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:25:40.550 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:25:40.384705Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:25:40.384705Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:25:40.384705Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:25:40.384705Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:40.384705Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:40.384705Z", "running": 0, "size": 1}}] 2022-03-08T23:25:50.574 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:25:51.361 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:25:51.361 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:25:51.116477Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:25:51.116477Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:25:51.116477Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:25:51.116477Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:25:51.116477Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:25:51.116477Z", "running": 0, "size": 1}}] 2022-03-08T23:26:01.389 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:26:02.095 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:26:02.096 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:26:01.881630Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:26:01.881630Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:26:01.881630Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:26:01.881630Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:01.881630Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:01.881630Z", "running": 0, "size": 1}}] 2022-03-08T23:26:12.119 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:26:12.797 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:26:12.798 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:26:12.603983Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:26:12.603983Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:26:12.603983Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:26:12.603983Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:12.603983Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:12.603983Z", "running": 0, "size": 1}}] 2022-03-08T23:26:22.822 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:26:23.551 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:26:23.552 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:26:23.334917Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:26:23.334917Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:26:23.334917Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:26:23.334917Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:23.334917Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:23.334917Z", "running": 0, "size": 1}}] 2022-03-08T23:26:33.579 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:26:34.258 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:26:34.258 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:26:34.045800Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:26:34.045800Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:26:34.045800Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:26:34.045800Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:34.045800Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:34.045800Z", "running": 0, "size": 1}}] 2022-03-08T23:26:44.282 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:26:45.009 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:26:45.010 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:26:44.795109Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:26:44.795109Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:26:44.795109Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:26:44.795109Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:44.795109Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:44.795109Z", "running": 0, "size": 1}}] 2022-03-08T23:26:55.034 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:26:55.799 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:26:55.799 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:26:55.616116Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:26:55.616116Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:26:55.616116Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:26:55.616116Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:26:55.616116Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:26:55.616116Z", "running": 0, "size": 1}}] 2022-03-08T23:27:00.885 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:27:00.883704 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:27:01.597 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:27:01.595980 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:27:01.585+0000 7f6c4699a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:27:01.598 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:27:01.585+0000 7f6c4699a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:27:01.598 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:27:01.585+0000 7f6c4699a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:27:01.598 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:27:05.820 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:27:06.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:27:06.539 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:27:06.335627Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:27:06.335627Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:27:06.335627Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:27:06.335627Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:06.335627Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:06.335627Z", "running": 0, "size": 1}}] 2022-03-08T23:27:16.564 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:27:17.244 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:27:17.244 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:27:17.051031Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:27:17.051031Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:27:17.051031Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:27:17.051031Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:17.051031Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:17.051031Z", "running": 0, "size": 1}}] 2022-03-08T23:27:27.268 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:27:27.959 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:27:27.960 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:27:27.767555Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:27:27.767555Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:27:27.767555Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:27:27.767555Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:27.767555Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:27.767555Z", "running": 0, "size": 1}}] 2022-03-08T23:27:37.986 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:27:38.764 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:27:38.765 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:27:38.582903Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:27:38.582903Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:27:38.582903Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:27:38.582903Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:38.582903Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:38.582903Z", "running": 0, "size": 1}}] 2022-03-08T23:27:48.793 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:27:49.538 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:27:49.538 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:27:49.271593Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:27:49.271593Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:27:49.271593Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:27:49.271593Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:27:49.271593Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:27:49.271593Z", "running": 0, "size": 1}}] 2022-03-08T23:27:59.567 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:28:00.355 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:28:00.356 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:28:00.057939Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:28:00.057939Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:28:00.057939Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:28:00.057939Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:00.057939Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:00.057939Z", "running": 0, "size": 1}}] 2022-03-08T23:28:10.378 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:28:11.048 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:28:11.049 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:28:10.862078Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:28:10.862078Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:28:10.862078Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:28:10.862078Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:10.862078Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:10.862078Z", "running": 0, "size": 1}}] 2022-03-08T23:28:21.073 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:28:21.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:28:21.806 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:28:21.593580Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:28:21.593580Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:28:21.593580Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:28:21.593580Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:21.593580Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:21.593580Z", "running": 0, "size": 1}}] 2022-03-08T23:28:31.831 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:28:32.604 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:28:32.604 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:28:32.336283Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:28:32.336283Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:28:32.336283Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:28:32.336283Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:32.336283Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:32.336283Z", "running": 0, "size": 1}}] 2022-03-08T23:28:42.630 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:28:43.620 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:28:43.620 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:28:43.109292Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:28:43.109292Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:28:43.109292Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:28:43.109292Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:43.109292Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:43.109292Z", "running": 0, "size": 1}}] 2022-03-08T23:28:53.649 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:28:54.404 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:28:54.404 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:28:54.231512Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:28:54.231512Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:28:54.231512Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:28:54.231512Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:28:54.231512Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:28:54.231512Z", "running": 0, "size": 1}}] 2022-03-08T23:29:04.427 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:29:05.135 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:29:05.135 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:29:04.929421Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:29:04.929421Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:29:04.929421Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:29:04.929421Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:04.929421Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:04.929421Z", "running": 0, "size": 1}}] 2022-03-08T23:29:15.162 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:29:15.952 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:29:15.952 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:29:15.729690Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:29:15.729690Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:29:15.729690Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:29:15.729690Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:15.729690Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:15.729690Z", "running": 0, "size": 1}}] 2022-03-08T23:29:25.978 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:29:26.713 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:29:26.714 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:29:26.540125Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:29:26.540125Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:29:26.540125Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:29:26.540125Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:26.540125Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:26.540125Z", "running": 0, "size": 1}}] 2022-03-08T23:29:36.739 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:29:37.519 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:29:37.519 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:29:37.336950Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:29:37.336950Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:29:37.336950Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:29:37.336950Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:37.336950Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:37.336950Z", "running": 0, "size": 1}}] 2022-03-08T23:29:47.545 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:29:48.271 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:29:48.271 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:29:48.043875Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:29:48.043875Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:29:48.043875Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:29:48.043875Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:48.043875Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:48.043875Z", "running": 0, "size": 1}}] 2022-03-08T23:29:58.300 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:29:59.011 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:29:59.011 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:29:58.819320Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:29:58.819320Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:29:58.819320Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:29:58.819320Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:29:58.819320Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:29:58.819320Z", "running": 0, "size": 1}}] 2022-03-08T23:30:09.038 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:30:09.776 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:30:09.777 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:30:09.563838Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:30:09.563838Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:30:09.563838Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:30:09.563838Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:09.563838Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:09.563838Z", "running": 0, "size": 1}}] 2022-03-08T23:30:19.800 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:30:20.530 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:30:20.531 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:30:20.353706Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:30:20.353706Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:30:20.353706Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:30:20.353706Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:20.353706Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:20.353706Z", "running": 0, "size": 1}}] 2022-03-08T23:30:25.337 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:30:25.335905 E | ceph-rbd-mirror-controller: failed to reconcile failed to detect running and desired ceph version: failed to detect ceph image version: failed to complete ceph version job: failed to run CmdReporter ceph-rbd-mirror-controller-detect-version successfully. failed waiting for results ConfigMap ceph-rbd-mirror-controller-detect-version. timed out waiting for results ConfigMap 2022-03-08T23:30:30.559 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:30:31.051780Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:30:31.051780Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:30:31.051780Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:30:31.051780Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:31.051780Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:31.051780Z", "running": 0, "size": 1}}] 2022-03-08T23:30:41.262 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:30:42.334 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:30:42.334 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:30:41.794329Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:30:41.794329Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:30:41.794329Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:30:41.794329Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:41.794329Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:41.794329Z", "running": 0, "size": 1}}] 2022-03-08T23:30:52.360 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:30:53.037 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:30:53.037 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:30:52.860226Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:30:52.860226Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:30:52.860226Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:30:52.860226Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:30:52.860226Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:30:52.860226Z", "running": 0, "size": 1}}] 2022-03-08T23:31:03.062 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:31:03.699 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:31:03.699 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:31:03.518805Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:31:03.518805Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:31:03.518805Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:31:03.518805Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:31:03.518805Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:31:03.518805Z", "running": 0, "size": 1}}] 2022-03-08T23:31:13.724 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:31:14.430 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:31:14.431 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:31:14.238977Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:31:14.238977Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:31:14.238977Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:31:14.238977Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:31:14.238977Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:31:14.238977Z", "running": 0, "size": 1}}] 2022-03-08T23:31:24.453 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:31:25.229 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:31:25.230 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:31:24.968457Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:31:24.968457Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:31:24.968457Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:31:24.968457Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:31:24.968457Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:31:24.968457Z", "running": 0, "size": 1}}] 2022-03-08T23:31:35.250 DEBUG:teuthology.orchestra.run.smithi107:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-zvbgc -- ceph orch ls -f json 2022-03-08T23:31:35.905 INFO:teuthology.orchestra.run.smithi107.stdout: 2022-03-08T23:31:35.906 INFO:teuthology.orchestra.run.smithi107.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:58.000000Z", "last_refresh": "2022-03-08T23:31:35.726761Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:09:23.000000Z", "last_refresh": "2022-03-08T23:31:35.726761Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:06:28.000000Z", "last_refresh": "2022-03-08T23:31:35.726761Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T23:10:59.000000Z", "last_refresh": "2022-03-08T23:31:35.726761Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T23:31:35.726761Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T23:31:35.726761Z", "running": 0, "size": 1}}] 2022-03-08T23:31:35.932 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:31:35.932 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:31:35.933 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-08T23:31:35.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:31:35.945 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:31:35.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T23:31:35.978 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:31:35.979 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:31:35.990 DEBUG:teuthology.orchestra.remote:smithi107:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T23:31:36.004 DEBUG:teuthology.orchestra.run.smithi107:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-08T23:31:36.070 INFO:teuthology.orchestra.run.smithi107.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-08T23:31:36.091 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T23:31:36.091 DEBUG:teuthology.orchestra.run.smithi107:> kubectl delete -f cluster.yaml 2022-03-08T23:31:36.249 INFO:teuthology.orchestra.run.smithi107.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-08T23:31:36.251 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.250772 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-08T23:31:36.251 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.250845 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T23:31:36.254 INFO:tasks.rook.operator.smithi107.stdout:I0308 23:31:36.251102 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T23:31:36.255 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.251835 I | operator: successfully started the controller-runtime manager 2022-03-08T23:31:36.272 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.271592 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T23:31:36.272 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.271621 I | operator: watching all namespaces for Ceph CRs 2022-03-08T23:31:36.273 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.271667 I | operator: setting up schemes 2022-03-08T23:31:36.280 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.274391 I | operator: setting up the controller-runtime manager 2022-03-08T23:31:36.579 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.578201 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T23:31:36.587 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.580856 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T23:31:36.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.580913 I | ceph-cluster-controller: successfully started 2022-03-08T23:31:36.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.580995 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T23:31:36.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581032 I | ceph-crashcollector-controller: successfully started 2022-03-08T23:31:36.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581066 I | ceph-block-pool-controller: successfully started 2022-03-08T23:31:36.589 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581095 I | ceph-object-store-user-controller: successfully started 2022-03-08T23:31:36.589 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581130 I | ceph-object-realm-controller: successfully started 2022-03-08T23:31:36.589 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581156 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T23:31:36.589 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581178 I | ceph-object-zone-controller: successfully started 2022-03-08T23:31:36.590 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581378 I | ceph-object-controller: successfully started 2022-03-08T23:31:36.590 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581444 I | ceph-file-controller: successfully started 2022-03-08T23:31:36.590 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581505 I | ceph-nfs-controller: successfully started 2022-03-08T23:31:36.590 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581554 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T23:31:36.590 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581606 I | ceph-client-controller: successfully started 2022-03-08T23:31:36.591 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581643 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T23:31:36.591 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581687 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T23:31:36.591 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581718 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T23:31:36.591 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581744 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T23:31:36.592 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581776 I | ceph-bucket-topic: successfully started 2022-03-08T23:31:36.592 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581797 I | ceph-bucket-notification: successfully started 2022-03-08T23:31:36.592 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581819 I | ceph-bucket-notification: successfully started 2022-03-08T23:31:36.592 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.581842 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T23:31:36.592 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.583106 I | operator: starting the controller-runtime manager 2022-03-08T23:31:36.687 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.686431 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T23:31:36.687 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.686533 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T23:31:36.688 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.686550 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T23:31:36.691 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.689907 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T23:31:36.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.694277 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:36.821 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.820965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:36.832 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:36.831938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:37.336 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:37.335704 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:37.325+0000 7f94f377b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:37.337 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:37.325+0000 7f94f377b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:37.337 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:37.325+0000 7f94f377b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:37.337 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:37.383 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:37.382671 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:38.067 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:38.065808 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:38.057+0000 7f06752b1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:38.067 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:38.057+0000 7f06752b1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:38.067 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:38.057+0000 7f06752b1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:38.067 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:38.092 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:38.091797 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:38.706 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:38.705817 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:38.697+0000 7f6c11ce7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:38.707 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:38.697+0000 7f6c11ce7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:38.707 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:38.697+0000 7f6c11ce7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:38.707 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:38.887 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:38.886093 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:39.572 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:39.571678 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:39.565+0000 7fa79f00d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:39.573 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:39.565+0000 7fa79f00d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:39.573 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:39.565+0000 7fa79f00d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:39.573 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:39.618 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:39.618034 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:40.255 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:40.254372 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:40.245+0000 7fbc81c3c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:40.255 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:40.245+0000 7fbc81c3c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:40.255 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:40.245+0000 7fbc81c3c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:40.256 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:40.342 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:40.341393 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:40.983 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:40.982072 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:40.965+0000 7fdf08a03380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:40.983 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:40.965+0000 7fdf08a03380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:40.983 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:40.965+0000 7fdf08a03380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:40.984 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:41.149 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:41.148229 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:41.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:41.946670 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:41.937+0000 7f3521cb2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:41.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:41.937+0000 7f3521cb2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:41.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:41.937+0000 7f3521cb2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:41.948 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:42.275 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:42.274573 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:43.025 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:43.024291 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:43.013+0000 7fc404058380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:43.025 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:43.013+0000 7fc404058380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:43.025 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:43.013+0000 7fc404058380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:43.026 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:43.671 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:43.670575 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:44.328 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:44.327003 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:44.293+0000 7f474876e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:44.328 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:44.293+0000 7f474876e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:44.328 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:44.293+0000 7f474876e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:44.328 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:45.612 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:45.611555 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:46.317 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:46.315943 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:46.305+0000 7f8779fca380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:46.317 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:46.305+0000 7f8779fca380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:46.317 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:46.305+0000 7f8779fca380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:46.318 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:46.924 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:46.924292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:46.973 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:46.972367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:48.883 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:48.882374 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:49.531 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:49.530341 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:49.521+0000 7f49596ae380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:49.531 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:49.521+0000 7f49596ae380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:49.532 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:49.521+0000 7f49596ae380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:49.532 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:54.660 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:54.659419 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:31:55.307 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:55.306038 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:31:55.297+0000 7fc863aab380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:31:55.307 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:55.297+0000 7fc863aab380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:31:55.307 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:31:55.297+0000 7fc863aab380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:31:55.308 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:31:57.033 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:57.033179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:31:57.046 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:31:57.045851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:05.552 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:05.551978 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:32:06.187 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:06.186615 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:32:06.177+0000 7fa64f618380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:32:06.188 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:32:06.177+0000 7fa64f618380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:32:06.188 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:32:06.177+0000 7fa64f618380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:32:06.188 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:32:07.534 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:07.533832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:07.545 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:07.545224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:17.653 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:17.652350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:17.665 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:17.664895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:26.674 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:26.673417 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:32:27.305 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:27.304041 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:32:27.288+0000 7f4703f1f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:32:27.305 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:32:27.288+0000 7f4703f1f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:32:27.305 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:32:27.288+0000 7f4703f1f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:32:27.305 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:32:27.979 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:27.979027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:27.991 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:27.991156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:38.083 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:38.082596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:38.095 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:38.094425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:48.206 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:48.205409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:48.217 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:48.217143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:58.315 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:58.314870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:32:58.332 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:32:58.332029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:08.272 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:08.270788 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:33:08.390 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:08.389325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:08.399 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:08.399233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:08.929 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:08.928658 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:33:08.920+0000 7fb491e95380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:33:08.930 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:33:08.920+0000 7fb491e95380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:33:08.930 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:33:08.920+0000 7fb491e95380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:33:08.930 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:33:18.467 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:18.466704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:18.479 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:18.478897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:28.647 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:28.646588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:28.661 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:28.660733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:38.716 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:38.713750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:38.725 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:38.725345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:48.827 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:48.827266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:48.840 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:48.839638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:59.002 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:59.001598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:33:59.014 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:33:59.014012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:09.124 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:09.123667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:09.136 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:09.136010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:19.234 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:19.233917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:19.245 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:19.244449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:29.405 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:29.404281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:29.418 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:29.418120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:30.861 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:30.860667 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:34:31.497 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:31.496141 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:34:31.482+0000 7f50d1981380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:34:31.497 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:34:31.482+0000 7f50d1981380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:34:31.497 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:34:31.482+0000 7f50d1981380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:34:31.498 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:34:39.536 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:39.535298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:39.549 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:39.548479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:49.612 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:49.612017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:49.667 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:49.666785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:59.763 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:59.762766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:34:59.775 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:34:59.774307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:09.853 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:09.853080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:09.865 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:09.865025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:19.975 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:19.974715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:19.986 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:19.985765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:30.113 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:30.113151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:30.127 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:30.126654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:40.226 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:40.226159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:40.238 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:40.237940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:50.344 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:50.343886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:35:50.356 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:35:50.356193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:00.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:00.414248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:00.429 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:00.428832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:10.512 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:10.512082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:10.604 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:10.603271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:20.813 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:20.812397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:20.867 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:20.867334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:30.938 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:30.937194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:30.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:30.947438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:41.088 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:41.088056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:41.101 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:41.101328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:51.155 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:51.154800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:36:51.166 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:36:51.165619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:01.303 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:01.303032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:01.315 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:01.315435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:11.395 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:11.395253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:11.414 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:11.413372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:15.344 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:15.343721 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:37:15.990 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:15.989360 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:37:15.980+0000 7f23d7a1a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:37:15.990 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:37:15.980+0000 7f23d7a1a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:37:15.990 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:37:15.980+0000 7f23d7a1a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:37:15.991 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:37:21.530 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:21.529703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:21.541 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:21.541016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:31.656 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:31.655751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:31.669 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:31.669156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:41.745 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:41.744954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:41.756 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:41.756031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:51.849 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:51.849256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:37:51.863 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:37:51.863189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:01.939 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:01.938289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:01.950 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:01.949755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:12.012 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:12.011931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:12.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:12.022660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:22.140 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:22.140269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:22.152 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:22.152144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:32.214 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:32.211016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:32.223 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:32.223065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:42.327 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:42.326451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:42.339 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:42.339630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:52.428 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:52.427718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:38:52.477 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:38:52.476518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:02.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:02.587458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:02.601 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:02.600754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:12.756 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:12.755616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:12.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:12.771263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:22.899 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:22.898659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:22.910 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:22.910431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:33.019 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:33.018752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:33.030 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:33.030095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:43.109 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:43.109382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:43.122 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:43.121562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:53.221 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:53.220899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:39:53.233 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:39:53.232712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:03.298 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:03.297842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:03.310 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:03.309891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:13.377 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:13.376745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:13.411 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:13.411223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:23.487 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:23.486696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:23.500 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:23.499917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:33.599 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:33.598521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:33.611 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:33.611023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:43.707 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:43.706281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:43.720 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:43.719784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:53.831 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:53.830532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:40:53.843 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:40:53.842792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:03.933 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:03.932909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:03.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:03.946206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:14.048 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:14.047865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:14.060 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:14.060057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:24.165 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:24.165034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:24.176 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:24.176283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:34.239 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:34.239262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:34.286 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:34.285742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:44.427 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:44.427102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:44.438 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:44.437978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:54.503 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:54.502882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:41:54.512 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:41:54.512312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:04.602 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:04.601646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:04.618 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:04.618386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:14.709 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:14.709488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:14.766 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:14.765814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:24.834 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:24.834238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:24.849 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:24.847394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:35.025 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:35.024707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:35.037 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:35.037503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:43.678 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:43.677773 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:42:44.307 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:44.306392 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:42:44.294+0000 7f0c691a4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:42:44.307 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:42:44.294+0000 7f0c691a4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:42:44.308 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:42:44.294+0000 7f0c691a4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:42:44.308 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:42:45.296 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:45.295515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:45.308 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:45.308176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:55.382 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:55.382301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:42:55.397 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:42:55.397101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:05.461 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:05.461186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:05.474 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:05.474254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:15.586 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:15.586245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:15.599 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:15.599545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:25.667 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:25.666860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:25.678 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:25.678013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:35.738 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:35.737937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:35.749 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:35.748803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:45.838 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:45.837465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:45.850 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:45.850114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:55.916 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:55.916410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:43:55.929 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:43:55.927870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:06.027 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:06.027341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:06.039 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:06.039162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:16.142 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:16.142063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:16.152 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:16.152283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:26.250 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:26.249996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:26.260 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:26.259475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:36.342 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:36.341993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:36.358 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:36.356381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:46.448 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:46.447645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:46.459 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:46.459380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:56.548 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:56.548401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:44:56.561 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:44:56.560727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:06.656 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:06.655610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:06.671 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:06.670332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:16.784 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:16.784205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:16.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:16.796381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:26.879 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:26.878909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:26.887 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:26.887444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:36.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:36.984675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:36.995 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:36.994937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:47.081 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:47.080948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:47.091 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:47.090667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:57.186 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:57.186346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:45:57.197 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:45:57.197363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:07.271 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:07.266611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:07.279 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:07.278674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:17.349 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:17.348659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:17.361 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:17.361579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:27.422 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:27.421859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:27.434 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:27.433808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:37.500 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:37.500147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:37.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:37.587665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:47.661 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:47.661355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:47.671 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:47.671525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:57.738 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:57.737648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:46:57.806 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:46:57.806159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:07.884 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:07.883899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:07.895 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:07.894976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:17.964 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:17.963856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:17.978 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:17.976945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:28.075 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:28.074966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:28.093 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:28.088553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:38.165 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:38.164966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:38.180 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:38.179621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:48.233 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:48.230727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:48.274 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:48.271909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:58.364 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:58.363566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:47:58.377 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:47:58.376918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:08.496 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:08.496143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:08.511 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:08.510661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:18.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:18.582938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:18.597 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:18.596656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:28.672 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:28.672220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:28.685 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:28.685276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:38.746 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:38.745967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:38.758 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:38.757889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:48.823 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:48.823277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:48.834 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:48.833884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:58.948 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:58.947399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:48:58.961 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:48:58.960201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:09.063 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:09.062562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:09.075 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:09.075129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:19.173 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:19.172628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:19.184 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:19.184115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:29.324 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:29.323772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:29.336 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:29.336300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:39.472 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:39.471817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:39.483 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:39.482265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:49.619 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:49.619059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:49.631 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:49.631552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:59.724 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:59.724359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:49:59.738 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:49:59.737891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:09.810 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:09.810110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:09.822 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:09.821030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:19.878 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:19.877800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:19.890 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:19.890306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:30.002 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:30.002204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:30.014 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:30.014456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:40.101 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:40.100173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:40.113 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:40.112665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:50.246 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:50.245303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:50:50.258 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:50:50.258299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:00.322 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:00.322028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:00.337 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:00.337067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:10.404 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:10.403507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:10.416 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:10.416144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:20.489 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:20.488477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:20.501 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:20.501399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:30.561 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:30.560728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:30.623 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:30.622403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:40.685 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:40.684949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:40.698 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:40.697766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:50.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:50.796153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:51:50.815 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:51:50.814815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:00.883 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:00.883204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:00.896 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:00.896315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:11.000 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:10.999931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:11.011 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:11.011003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:21.114 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:21.114270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:21.125 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:21.125102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:31.221 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:31.221034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:31.232 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:31.232026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:41.305 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:41.304490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:41.321 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:41.320945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:51.397 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:51.396335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:52:51.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:52:51.414615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:01.480 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:01.480180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:01.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:01.492957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:11.574 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:11.573461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:11.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:11.588138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:21.641 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:21.640941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:21.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:21.652007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:31.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:31.795921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:31.807 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:31.806455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:39.674 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:39.673895 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-08T23:53:40.335 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:40.334646 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T23:53:40.323+0000 7fde1b1fc380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T23:53:40.341 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:53:40.323+0000 7fde1b1fc380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T23:53:40.342 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08T23:53:40.323+0000 7fde1b1fc380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T23:53:40.342 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-08T23:53:41.894 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:41.894152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:41.957 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:41.956938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:52.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:52.167001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:53:52.178 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:53:52.176798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:02.289 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:02.288587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:02.300 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:02.300243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:12.369 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:12.368414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:12.382 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:12.381001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:22.490 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:22.488730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:22.502 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:22.501126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:32.570 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:32.569653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:32.581 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:32.581395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:42.644 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:42.644029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:42.656 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:42.656643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:52.722 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:52.721231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:54:52.773 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:54:52.772269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:02.840 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:02.839891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:02.851 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:02.850631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:12.961 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:12.960910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:12.971 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:12.971136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:23.141 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:23.140550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:23.156 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:23.155461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:33.256 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:33.255487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:33.268 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:33.268427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:43.388 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:43.387698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:43.400 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:43.399712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:53.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:53.493736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:55:53.506 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:55:53.506545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:03.608 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:03.608072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:03.621 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:03.620639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:13.716 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:13.715816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:13.728 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:13.728784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:23.828 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:23.828301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:23.839 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:23.839379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:33.925 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:33.925148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:33.938 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:33.937903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:44.040 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:44.040291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:44.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:44.053205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:54.124 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:54.124128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:56:54.136 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:56:54.135741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:04.230 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:04.230375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:04.241 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:04.240902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:14.358 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:14.357541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:14.370 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:14.369901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:24.439 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:24.438568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:24.493 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:24.493297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:34.554 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:34.554102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:34.566 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:34.565961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:44.675 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:44.674133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:44.688 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:44.688196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:54.759 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:54.758261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:57:54.773 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:57:54.773124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:05.033 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:05.033507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:05.042 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:05.042602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:15.197 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:15.196184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:15.207 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:15.207029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:25.304 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:25.304313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:25.318 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:25.318009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:35.373 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:35.373046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:35.388 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:35.387761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:45.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:45.492192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:45.503 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:45.503235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:55.629 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:55.628607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:58:55.643 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:58:55.642920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:05.708 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:05.707953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:05.719 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:05.719177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:15.816 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:15.815588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:15.828 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:15.827940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:25.920 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:25.919239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:25.966 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:25.964632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:36.041 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:36.040548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:36.053 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:36.053243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:46.126 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:46.121183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:46.133 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:46.133625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:56.269 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:56.269212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T23:59:56.280 INFO:tasks.rook.operator.smithi107.stdout:2022-03-08 23:59:56.280538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:06.339 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:06.338936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:06.393 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:06.392531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:16.542 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:16.541781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:16.554 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:16.553850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:26.625 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:26.625026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:26.636 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:26.635718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:36.744 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:36.744130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:36.757 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:36.757189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:46.822 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:46.821792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:46.899 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:46.898384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:57.014 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:57.013957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:00:57.027 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:00:57.027177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:07.101 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:07.101137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:07.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:07.112088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:17.200 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:17.200043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:17.218 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:17.218555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:27.308 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:27.307510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:27.322 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:27.322406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:37.456 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:37.455605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:37.465 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:37.465395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:47.527 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:47.524951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:47.536 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:47.536045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:57.645 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:57.645003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:01:57.655 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:01:57.654874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:07.713 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:07.712694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:07.742 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:07.732299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:17.861 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:17.860908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:17.872 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:17.872283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:27.971 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:27.970540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:27.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:27.982258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:38.159 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:38.159191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:38.172 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:38.172031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:48.256 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:48.256509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:48.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:48.269979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:58.407 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:58.406433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:02:58.418 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:02:58.417635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:08.522 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:08.521535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:08.533 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:08.532921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:18.620 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:18.619666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:18.633 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:18.632938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:28.705 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:28.705303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:28.718 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:28.717653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:38.822 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:38.822025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:38.833 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:38.832928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:48.896 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:48.895139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:48.907 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:48.907483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:59.013 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:59.013454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:03:59.024 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:03:59.023307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:09.091 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:09.090996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:09.103 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:09.103132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:19.204 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:19.204036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:19.215 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:19.215384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:29.322 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:29.321352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:29.338 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:29.334847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:39.411 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:39.410969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:39.423 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:39.423318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:49.537 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:49.536387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:49.548 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:49.548502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:59.628 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:59.627556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:04:59.640 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:04:59.639745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:09.747 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:09.747411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:09.760 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:09.760079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:19.831 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:19.830875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:19.845 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:19.845013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:29.907 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:29.906692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:29.919 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:29.919491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:40.031 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:40.030710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:40.043 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:40.043184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:50.145 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:50.145098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:05:50.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:05:50.156685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:00.216 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:00.215498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:00.227 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:00.227254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:10.322 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:10.322189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:10.371 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:10.371010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:20.440 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:20.439691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:20.487 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:20.486798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:30.551 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:30.551170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:30.594 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:30.594318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:40.658 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:40.658261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:40.670 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:40.669606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:50.783 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:50.782676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:06:50.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:06:50.795530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:00.885 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:00.884797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:00.899 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:00.898908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:10.987 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:10.987153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:11.000 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:10.999639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:21.143 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:21.142516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:21.155 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:21.155357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:31.246 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:31.245755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:31.258 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:31.257875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:41.368 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:41.367543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:41.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:41.377918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:51.486 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:51.485906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:07:51.499 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:07:51.499278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:01.605 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:01.604559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:01.615 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:01.615588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:11.684 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:11.683975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:11.739 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:11.739296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:21.853 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:21.853197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:21.865 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:21.864905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:32.052 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:32.051237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:32.065 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:32.064692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:42.208 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:42.207940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:42.222 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:42.221483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:52.285 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:52.284597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:08:52.303 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:08:52.303281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:02.374 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:02.374176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:02.388 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:02.387313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:12.453 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:12.452690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:12.513 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:12.512800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:22.581 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:22.581186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:22.594 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:22.594230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:32.686 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:32.685658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:32.697 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:32.697476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:42.808 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:42.807425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:42.819 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:42.818916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:52.887 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:52.887091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:09:52.960 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:09:52.959407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:03.017 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:03.017023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:03.028 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:03.027903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:13.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:13.156952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:13.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:13.166493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:20.340 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:20.340361 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T00:10:20.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:20.982921 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:10:20.974+0000 7f8012609380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:10:20.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T00:10:20.974+0000 7f8012609380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:10:20.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T00:10:20.974+0000 7f8012609380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:10:20.985 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:10:23.271 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:23.270443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:23.283 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:23.283012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:33.349 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:33.348425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:33.360 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:33.360397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:43.425 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:43.424457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:43.437 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:43.436706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:53.527 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:53.526180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:10:53.540 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:10:53.539662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:03.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:03.650360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:03.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:03.662498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:13.720 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:13.719628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:13.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:13.732288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:23.809 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:23.808845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:23.822 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:23.821521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:33.962 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:33.961108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:33.974 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:33.972362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:44.047 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:44.042675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:44.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:44.053878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:54.113 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:54.112216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:11:54.211 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:11:54.210120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:04.294 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:04.292852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:04.370 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:04.370188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:14.482 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:14.481644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:14.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:14.494323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:24.560 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:24.559270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:24.572 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:24.571607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:34.661 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:34.660145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:34.673 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:34.673246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:44.739 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:44.738382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:44.751 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:44.750842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:54.825 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:54.824498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:12:54.839 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:12:54.837800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:04.902 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:04.901771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:04.915 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:04.914339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:14.992 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:14.991999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:15.048 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:15.047481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:25.151 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:25.150959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:25.163 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:25.162699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:35.234 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:35.233124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:35.303 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:35.302622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:45.360 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:45.359371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:45.413 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:45.412411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:55.472 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:55.472080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:13:55.483 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:13:55.482977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:05.575 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:05.574868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:05.585 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:05.585033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:15.693 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:15.692202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:15.706 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:15.705505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:25.804 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:25.804169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:25.817 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:25.816634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:35.873 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:35.872416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:35.887 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:35.886511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:46.047 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:46.046328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:46.063 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:46.062828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:56.173 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:56.172493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:14:56.183 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:14:56.183266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:06.305 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:06.304720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:06.315 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:06.315314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:16.408 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:16.407259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:16.417 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:16.416993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:26.485 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:26.484318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:26.497 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:26.496997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:36.568 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:36.567193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:36.582 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:36.580838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:46.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:46.650702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:46.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:46.694582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:56.759 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:56.758632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:15:56.782 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:15:56.781344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:06.889 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:06.889006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:06.903 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:06.902269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:17.016 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:17.015603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:17.025 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:17.025096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:27.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:27.132232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:27.144 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:27.144294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:37.267 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:37.266646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:37.282 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:37.281605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:47.362 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:47.361605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:47.411 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:47.410978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:57.507 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:57.506373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:16:57.517 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:16:57.516907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:07.609 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:07.609162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:07.622 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:07.621691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:17.698 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:17.698104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:17.711 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:17.710993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:27.808 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:27.807411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:27.821 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:27.821167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:37.902 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:37.901203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:37.914 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:37.913678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:47.987 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:47.986781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:48.070 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:48.069805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:58.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:58.157183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:17:58.174 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:17:58.172957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:08.235 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:08.235122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:08.249 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:08.248632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:18.344 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:18.343548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:18.357 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:18.356570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:28.469 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:28.468623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:28.481 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:28.481383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:38.575 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:38.575014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:38.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:38.587771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:48.691 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:48.690092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:48.704 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:48.703504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:58.765 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:58.764873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:18:58.777 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:18:58.776883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:08.892 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:08.892005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:08.903 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:08.902300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:18.961 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:18.960922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:18.974 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:18.973813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:29.047 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:29.047033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:29.063 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:29.062423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:39.169 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:39.168870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:39.181 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:39.180942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:49.252 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:49.251334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:49.267 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:49.266708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:59.374 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:59.373460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:19:59.389 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:19:59.388715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:09.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:09.454770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:09.470 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:09.469578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:19.530 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:19.530064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:19.584 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:19.583863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:29.679 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:29.678378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:29.692 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:29.691627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:39.789 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:39.788444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:39.802 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:39.801463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:49.872 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:49.871996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:49.885 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:49.884796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:59.970 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:59.969693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:20:59.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:20:59.983594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:10.056 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:10.055920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:10.067 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:10.066886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:20.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:20.132093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:20.145 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:20.145049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:30.222 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:30.221353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:30.234 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:30.233922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:40.338 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:40.337346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:40.346 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:40.346191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:50.471 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:50.470621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:21:50.482 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:21:50.481890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:00.568 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:00.567296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:00.580 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:00.579049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:10.669 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:10.668297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:10.683 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:10.682562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:20.763 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:20.762127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:20.775 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:20.774563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:30.929 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:30.928668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:30.938 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:30.937845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:41.061 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:41.061122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:41.074 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:41.073817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:51.166 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:51.165505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:22:51.181 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:22:51.180871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:01.251 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:01.250962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:01.264 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:01.263956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:11.365 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:11.364778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:11.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:11.377279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:21.476 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:21.475819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:21.488 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:21.488242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:31.587 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:31.586547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:31.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:31.599545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:41.673 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:41.672210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:41.687 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:41.686925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:51.794 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:51.793147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:23:51.806 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:23:51.806055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:01.875 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:01.874446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:01.887 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:01.887125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:12.000 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:11.999537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:12.013 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:12.012821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:22.068 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:22.067920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:22.080 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:22.079913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:32.180 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:32.180026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:32.195 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:32.192339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:42.284 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:42.283305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:42.296 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:42.295888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:52.383 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:52.382894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:24:52.393 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:24:52.392934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:02.495 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:02.494337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:02.510 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:02.509523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:12.628 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:12.628205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:12.640 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:12.639899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:22.744 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:22.744074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:22.762 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:22.759173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:32.870 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:32.869860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:32.882 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:32.881767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:42.993 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:42.991891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:43.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:43.004330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:53.076 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:53.075029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:25:53.088 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:25:53.087609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:03.156 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:03.155188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:03.169 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:03.168805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:13.241 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:13.240647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:13.255 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:13.254583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:23.327 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:23.326699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:23.343 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:23.342648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:33.401 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:33.400368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:33.413 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:33.413029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:43.517 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:43.516034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:43.531 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:43.530174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:53.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:53.638375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:26:53.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:26:53.650271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:01.011 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:01.011128 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T00:27:01.651 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:01.650722 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:27:01.641+0000 7f3dde5fa380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:27:01.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T00:27:01.641+0000 7f3dde5fa380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:27:01.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T00:27:01.641+0000 7f3dde5fa380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:27:01.652 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:27:03.745 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:03.744973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:03.758 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:03.758128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:13.877 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:13.877033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:13.889 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:13.888964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:23.998 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:23.997634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:24.011 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:24.010906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:34.081 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:34.080740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:34.094 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:34.093728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:44.164 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:44.164084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:44.177 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:44.176599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:54.248 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:54.248337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:27:54.263 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:27:54.263007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:04.332 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:04.331738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:04.346 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:04.346266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:14.405 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:14.405011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:14.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:14.415265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:24.508 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:24.508205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:24.522 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:24.521899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:34.623 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:34.622393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:34.636 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:34.635288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:44.706 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:44.705970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:44.720 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:44.719352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:54.830 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:54.829297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:28:54.841 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:28:54.840659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:04.942 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:04.942184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:04.955 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:04.954965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:15.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:15.053334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:15.068 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:15.067951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:25.136 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:25.135139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:25.148 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:25.147346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:35.228 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:35.227098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:35.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:35.269767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:45.336 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:45.335116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:45.351 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:45.350655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:55.419 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:55.418759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:29:55.444 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:29:55.443394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:05.547 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:05.546938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:05.559 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:05.559060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:15.658 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:15.657860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:15.672 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:15.672355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:25.768 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:25.767402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:25.780 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:25.780237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:35.854 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:35.853582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:35.866 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:35.865856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:45.927 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:45.926436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:45.939 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:45.939109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:56.044 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:56.043765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:30:56.099 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:30:56.097869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:06.226 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:06.225345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:06.236 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:06.235572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:16.297 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:16.296773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:16.313 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:16.311820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:26.381 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:26.381347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:26.393 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:26.393259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:36.481 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:36.480515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:36.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:36.493723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:46.555 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:46.555164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:46.568 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:46.567993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:56.630 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:56.629965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:31:56.643 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:31:56.642523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:06.749 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:06.748619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:06.759 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:06.758448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:16.849 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:16.848397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:16.860 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:16.859751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:26.954 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:26.953621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:26.966 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:26.965816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:37.065 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:37.064485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:37.079 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:37.078378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:47.192 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:47.191772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:47.204 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:47.203396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:57.261 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:57.260276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:32:57.329 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:32:57.328601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:07.387 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:07.386780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:07.397 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:07.396097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:17.466 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:17.465525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:17.526 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:17.525803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:27.805 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:27.805035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:27.894 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:27.893243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:37.958 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:37.958167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:37.969 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:37.969202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:48.080 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:48.080193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:48.088 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:48.088537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:58.222 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:58.221838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:33:58.235 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:33:58.234657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:08.303 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:08.303090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:08.315 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:08.314991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:18.401 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:18.400333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:18.412 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:18.411825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:28.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:28.491859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:28.557 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:28.557199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:38.629 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:38.628413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:38.641 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:38.640898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:48.703 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:48.703089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:48.713 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:48.712800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:58.778 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:58.778047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:34:58.789 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:34:58.789391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:08.932 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:08.932380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:08.942 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:08.942061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:19.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:19.004253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:19.017 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:19.016626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:29.082 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:29.081568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:29.093 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:29.093181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:39.204 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:39.203706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:39.217 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:39.216429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:49.272 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:49.271675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:49.335 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:49.335034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:59.402 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:59.401667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:35:59.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:35:59.414648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:09.556 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:09.556079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:09.569 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:09.568930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:19.672 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:19.671871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:19.683 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:19.683164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:29.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:29.770122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:29.785 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:29.785307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:39.915 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:39.914761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:39.925 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:39.925024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:50.046 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:50.045572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:36:50.059 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:36:50.059016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:00.180 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:00.179555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:00.188 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:00.188498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:10.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:10.258851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:10.272 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:10.271496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:20.384 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:20.383938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:20.397 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:20.396422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:30.490 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:30.489579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:30.505 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:30.504446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:40.574 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:40.573341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:40.585 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:40.584830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:50.658 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:50.657467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:37:50.669 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:37:50.669059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:00.745 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:00.744950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:00.757 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:00.756773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:10.827 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:10.827083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:10.836 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:10.835629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:20.956 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:20.955218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:20.968 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:20.967926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:31.029 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:31.029204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:31.042 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:31.041951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:41.145 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:41.144304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:41.156 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:41.156330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:51.232 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:51.231480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:38:51.287 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:38:51.287273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:01.390 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:01.390141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:01.402 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:01.402286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:11.509 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:11.509071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:11.520 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:11.519741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:21.619 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:21.619288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:21.631 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:21.630952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:31.696 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:31.696008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:31.707 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:31.706320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:41.869 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:41.868849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:41.881 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:41.881172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:51.950 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:51.948935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:39:51.962 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:39:51.961454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:02.081 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:02.080268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:02.093 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:02.092661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:12.235 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:12.234803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:12.245 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:12.245045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:22.317 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:22.315914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:22.329 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:22.328530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:32.406 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:32.406096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:32.416 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:32.416494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:42.546 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:42.546124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:42.557 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:42.556736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:52.646 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:52.645547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:40:52.658 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:40:52.658200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:02.757 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:02.757044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:02.769 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:02.769013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:12.860 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:12.859432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:12.874 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:12.873465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:22.971 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:22.970793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:22.983 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:22.982822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:33.094 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:33.093961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:33.106 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:33.105678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:43.175 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:43.174321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:43.187 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:43.187390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:53.328 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:53.327863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:41:53.340 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:41:53.340135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:03.435 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:03.435012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:03.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:03.454332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:13.540 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:13.540236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:13.554 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:13.554354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:23.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:23.651828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:23.665 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:23.664318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:33.783 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:33.782795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:33.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:33.794995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:43.926 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:43.925482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:43.937 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:43.937403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:54.038 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:54.037564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:42:54.052 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:42:54.052580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:04.149 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:04.149043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:04.163 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:04.162904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:14.286 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:14.285043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:14.298 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:14.297354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:24.400 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:24.399462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:24.410 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:24.410156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:34.497 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:34.496667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:34.556 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:34.556137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:41.660 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:41.660314 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T00:43:42.551 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:42.549943 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T00:43:42.540+0000 7f4e65a87380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T00:43:42.551 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T00:43:42.540+0000 7f4e65a87380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T00:43:42.551 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T00:43:42.540+0000 7f4e65a87380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T00:43:42.552 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T00:43:44.722 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:44.721245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:44.731 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:44.731507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:54.797 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:54.796668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:43:54.810 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:43:54.810184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:04.898 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:04.898004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:04.911 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:04.911125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:14.989 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:14.988622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:14.999 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:14.999561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:25.058 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:25.057852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:25.069 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:25.069344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:35.140 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:35.139361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:35.154 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:35.153701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:45.215 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:45.214290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:45.229 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:45.229261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:55.299 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:55.298946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:44:55.312 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:44:55.310727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:05.388 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:05.387877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:05.399 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:05.399375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:15.458 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:15.457864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:15.467 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:15.467267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:25.571 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:25.571161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:25.582 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:25.581990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:35.689 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:35.689470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:35.705 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:35.705401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:45.775 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:45.775346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:45.788 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:45.788176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:55.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:55.854281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:45:55.866 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:45:55.866171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:05.964 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:05.963621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:05.976 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:05.975686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:16.037 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:16.037196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:16.053 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:16.053272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:26.264 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:26.263604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:26.276 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:26.276423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:36.433 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:36.432938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:36.446 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:36.445883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:46.566 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:46.565143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:46.575 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:46.575290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:56.672 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:56.671890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:46:56.686 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:46:56.683980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:06.821 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:06.821269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:06.833 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:06.833546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:16.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:16.946274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:16.959 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:16.959392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:27.082 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:27.081515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:27.093 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:27.092995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:37.166 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:37.165756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:37.177 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:37.177286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:47.290 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:47.289521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:47.302 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:47.301829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:57.370 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:57.370269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:47:57.428 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:47:57.428062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:07.548 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:07.547964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:07.560 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:07.560186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:17.625 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:17.624698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:17.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:17.639139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:27.724 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:27.723557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:27.737 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:27.737207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:37.901 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:37.900868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:37.913 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:37.912772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:48.007 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:48.006362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:48.017 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:48.016930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:58.101 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:58.100760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:48:58.113 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:48:58.112707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:08.213 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:08.213275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:08.225 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:08.224577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:18.326 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:18.325990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:18.340 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:18.339811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:28.439 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:28.439348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:28.453 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:28.452774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:38.539 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:38.538988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:38.549 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:38.549301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:48.661 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:48.661191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:48.674 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:48.674072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:58.781 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:58.780966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:49:58.793 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:49:58.793728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:08.934 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:08.933846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:08.943 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:08.943314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:19.036 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:19.035332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:19.048 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:19.047810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:29.124 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:29.123321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:29.135 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:29.135677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:39.203 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:39.202791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:39.245 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:39.245244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:49.312 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:49.311898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:49.324 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:49.323905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:59.392 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:59.391223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:50:59.405 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:50:59.402628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:09.473 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:09.473203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:09.486 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:09.486336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:19.548 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:19.548326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:19.629 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:19.628963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:29.718 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:29.718032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:29.733 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:29.732780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:39.828 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:39.828341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:39.841 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:39.840780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:49.962 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:49.962057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:51:49.972 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:51:49.971975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:00.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:00.112270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:00.121 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:00.121164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:10.196 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:10.192853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:10.206 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:10.206302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:20.347 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:20.347177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:20.360 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:20.359710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:30.474 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:30.473728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:30.486 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:30.485920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:40.578 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:40.578206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:40.592 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:40.591825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:50.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:50.652486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:52:50.665 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:52:50.665502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:00.784 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:00.784009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:00.847 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:00.847583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:10.924 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:10.924462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:10.936 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:10.936413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:21.046 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:21.046353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:21.059 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:21.058745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:31.181 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:31.181382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:31.191 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:31.191332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:41.277 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:41.276856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:41.288 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:41.288312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:51.387 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:51.386837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:53:51.409 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:53:51.408855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:01.504 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:01.504327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:01.516 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:01.515975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:11.619 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:11.619254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:11.630 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:11.630087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:21.693 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:21.693069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:21.762 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:21.762534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:31.818 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:31.817829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:31.850 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:31.844278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:41.935 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:41.934982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:41.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:41.947249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:52.009 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:52.008958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:54:52.020 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:54:52.020074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:02.128 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:02.127621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:02.140 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:02.140086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:12.277 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:12.276376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:12.292 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:12.292435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:22.389 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:22.389319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:22.409 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:22.408714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:32.520 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:32.520618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:32.529 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:32.529337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:42.632 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:42.632085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:42.642 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:42.641845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:52.696 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:52.694832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:55:52.733 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:55:52.733171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:02.842 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:02.842255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:02.854 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:02.853182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:12.916 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:12.916244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:12.927 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:12.926919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:22.989 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:22.989205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:23.000 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:23.000190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:33.101 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:33.101195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:33.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:33.112521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:43.225 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:43.224875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:43.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:43.235615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:53.336 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:53.336181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:56:53.349 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:56:53.348534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:03.454 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:03.454119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:03.466 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:03.466603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:13.568 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:13.567660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:13.581 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:13.580179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:23.679 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:23.678435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:23.688 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:23.688595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:33.759 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:33.758547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:33.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:33.770748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:43.888 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:43.887544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:43.939 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:43.939582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:54.046 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:54.045656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:57:54.057 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:57:54.057660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:04.130 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:04.130135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:04.142 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:04.142572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:14.250 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:14.249823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:14.262 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:14.261765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:24.343 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:24.343341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:24.354 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:24.354239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:34.409 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:34.408500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:34.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:34.455511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:44.515 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:44.515656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:44.575 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:44.574739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:54.643 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:54.643151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:58:54.657 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:58:54.656648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:04.768 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:04.767943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:04.780 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:04.780319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:14.842 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:14.841898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:14.852 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:14.851831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:25.068 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:25.068369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:25.080 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:25.080200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:35.241 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:35.240712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:35.252 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:35.251853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:45.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:45.332846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:45.348 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:45.346493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:55.419 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:55.418918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T00:59:55.431 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 00:59:55.430836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:05.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:05.493512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:05.505 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:05.505128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:15.628 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:15.628045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:15.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:15.638890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:22.559 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:22.558111 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T01:00:23.215 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:23.214147 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:00:23.203+0000 7f1845cdf380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:00:23.216 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T01:00:23.203+0000 7f1845cdf380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:00:23.216 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T01:00:23.203+0000 7f1845cdf380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:00:23.216 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:00:25.723 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:25.722999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:25.772 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:25.771370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:35.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:35.854456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:35.869 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:35.868958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:45.962 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:45.961583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:45.974 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:45.974701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:56.048 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:56.048277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:00:56.061 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:00:56.060992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:06.177 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:06.176427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:06.189 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:06.189190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:16.313 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:16.313070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:16.327 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:16.327674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:26.394 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:26.393933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:26.407 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:26.406632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:36.512 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:36.511471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:36.524 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:36.524622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:46.618 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:46.617633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:46.630 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:46.630037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:56.710 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:56.710367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:01:56.792 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:01:56.792183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:06.934 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:06.933777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:06.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:06.946261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:17.008 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:17.007943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:17.020 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:17.019696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:27.123 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:27.123403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:27.135 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:27.135265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:37.276 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:37.275540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:37.286 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:37.286398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:47.373 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:47.373417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:47.386 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:47.386018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:57.475 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:57.474679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:02:57.487 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:02:57.487663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:07.544 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:07.541349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:07.553 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:07.553350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:17.628 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:17.628340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:17.641 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:17.641273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:27.699 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:27.697542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:27.709 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:27.709317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:37.811 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:37.810717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:37.823 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:37.822942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:47.908 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:47.907773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:47.920 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:47.920418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:58.030 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:58.030333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:03:58.042 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:03:58.041178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:08.135 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:08.135065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:08.146 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:08.146535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:18.240 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:18.239689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:18.253 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:18.252961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:28.348 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:28.347740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:28.361 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:28.361166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:38.426 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:38.426034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:38.438 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:38.437926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:48.502 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:48.501593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:48.515 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:48.514864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:58.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:58.582766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:04:58.653 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:04:58.652767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:08.753 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:08.752840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:08.769 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:08.768847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:18.869 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:18.868983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:18.884 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:18.884660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:28.943 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:28.943079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:29.001 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:29.000930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:39.161 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:39.160739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:39.173 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:39.173051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:49.264 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:49.264002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:49.283 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:49.283760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:59.348 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:59.347931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:05:59.360 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:05:59.360646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:09.439 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:09.438981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:09.451 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:09.451165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:19.537 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:19.536740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:19.551 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:19.550747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:29.661 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:29.661286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:29.675 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:29.675339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:39.751 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:39.751181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:39.813 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:39.813593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:49.920 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:49.919694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:06:49.934 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:06:49.934691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:00.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:00.004938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:00.019 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:00.018522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:10.096 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:10.096126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:10.108 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:10.108138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:20.169 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:20.169619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:20.257 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:20.256914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:30.376 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:30.376208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:30.389 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:30.388262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:40.479 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:40.479371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:40.495 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:40.495687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:50.567 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:50.566526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:07:50.606 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:07:50.605983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:00.670 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:00.669788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:00.721 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:00.721218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:10.836 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:10.835163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:10.846 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:10.846516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:20.961 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:20.960779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:20.973 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:20.973238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:31.086 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:31.085993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:31.097 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:31.097133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:41.211 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:41.211148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:41.223 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:41.223007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:51.282 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:51.282393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:08:51.296 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:08:51.296206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:01.391 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:01.390938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:01.403 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:01.402951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:11.465 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:11.464604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:11.478 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:11.478085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:21.550 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:21.550221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:21.601 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:21.600880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:31.685 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:31.685035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:31.696 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:31.696773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:41.758 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:41.758249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:41.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:41.771149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:51.875 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:51.874597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:09:51.886 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:09:51.886593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:02.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:02.004730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:02.019 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:02.019362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:12.117 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:12.116774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:12.130 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:12.130085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:22.195 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:22.192121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:22.208 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:22.208404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:32.290 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:32.289861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:32.301 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:32.301074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:42.372 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:42.372134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:42.426 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:42.426375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:52.511 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:52.510896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:10:52.566 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:10:52.566341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:02.626 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:02.626135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:02.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:02.639554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:12.725 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:12.724766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:12.737 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:12.735753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:22.845 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:22.845344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:22.858 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:22.857916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:32.943 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:32.942706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:32.953 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:32.953494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:43.052 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:43.052084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:43.064 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:43.064718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:53.129 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:53.128912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:11:53.141 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:11:53.141136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:03.230 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:03.229984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:03.242 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:03.242527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:13.335 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:13.334410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:13.347 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:13.347745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:23.498 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:23.498442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:23.512 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:23.512443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:33.611 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:33.610743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:33.621 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:33.620548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:43.727 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:43.727085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:43.795 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:43.794555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:53.856 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:53.856366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:12:53.870 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:12:53.869746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:03.989 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:03.989080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:04.001 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:04.000834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:14.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:14.053951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:14.066 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:14.065910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:24.137 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:24.136869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:24.148 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:24.148177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:34.251 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:34.251382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:34.266 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:34.264607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:44.379 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:44.379034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:44.393 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:44.391553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:54.459 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:54.459540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:13:54.471 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:13:54.470421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:04.526 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:04.526527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:04.544 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:04.544013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:14.656 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:14.655823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:14.666 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:14.666313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:24.742 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:24.741785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:24.803 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:24.803215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:34.865 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:34.864765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:34.880 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:34.879996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:45.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:45.005578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:45.018 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:45.018053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:55.081 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:55.080476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:14:55.092 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:14:55.092341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:05.222 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:05.221958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:05.230 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:05.230760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:15.290 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:15.290613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:15.301 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:15.301782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:25.410 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:25.409539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:25.422 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:25.421970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:35.490 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:35.489637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:35.503 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:35.503465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:45.569 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:45.569383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:45.626 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:45.626234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:55.697 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:55.697114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:15:55.709 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:15:55.709614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:05.774 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:05.774384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:05.843 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:05.843402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:15.954 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:15.953698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:15.964 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:15.963827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:26.088 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:26.088033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:26.100 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:26.099883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:36.231 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:36.231244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:36.243 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:36.243507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:46.323 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:46.322584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:46.337 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:46.336444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:56.416 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:56.416592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:16:56.488 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:16:56.488509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:03.223 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:03.222921 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T01:17:03.881 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:03.880820 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:17:03.870+0000 7f308c0b1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:17:03.881 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T01:17:03.870+0000 7f308c0b1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:17:03.882 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T01:17:03.870+0000 7f308c0b1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:17:03.882 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:17:06.559 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:06.558929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:06.568 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:06.568305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:16.686 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:16.685611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:16.699 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:16.699500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:26.799 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:26.798617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:26.810 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:26.810344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:36.881 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:36.881436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:36.894 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:36.894669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:46.990 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:46.990089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:47.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:47.004156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:57.066 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:57.065819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:17:57.077 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:17:57.076791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:07.141 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:07.140505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:07.207 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:07.152181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:17.223 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:17.222533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:17.236 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:17.236291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:27.294 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:27.293552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:27.305 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:27.305288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:37.390 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:37.390455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:37.402 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:37.402509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:47.454 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:47.453536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:47.465 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:47.465885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:57.547 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:57.547215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:18:57.562 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:18:57.561669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:07.630 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:07.629662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:07.643 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:07.642864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:17.737 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:17.736641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:17.750 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:17.749970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:27.896 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:27.896484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:27.909 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:27.908712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:38.039 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:38.038605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:38.048 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:38.048656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:48.111 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:48.111292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:48.125 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:48.125116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:58.186 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:58.186636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:19:58.199 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:19:58.199164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:08.254 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:08.253444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:08.265 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:08.264895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:18.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:18.355170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:18.367 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:18.367348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:28.452 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:28.452550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:28.478 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:28.478581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:38.552 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:38.552382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:38.563 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:38.563754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:48.700 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:48.700177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:48.713 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:48.713693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:58.817 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:58.817162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:20:58.827 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:20:58.827238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:08.890 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:08.890448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:08.903 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:08.902830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:18.959 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:18.959547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:18.969 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:18.969565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:29.040 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:29.040192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:29.052 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:29.051976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:39.123 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:39.123388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:39.201 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:39.135139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:49.235 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:49.234715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:49.248 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:49.248140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:59.342 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:59.341347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:21:59.352 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:21:59.351959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:09.450 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:09.449924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:09.462 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:09.462519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:19.531 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:19.531560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:19.544 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:19.544501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:29.607 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:29.606762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:29.645 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:29.645039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:39.750 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:39.749879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:39.759 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:39.760058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:49.829 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:49.829100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:49.842 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:49.841934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:59.952 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:59.952442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:22:59.961 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:22:59.961364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:10.066 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:10.065776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:10.079 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:10.079104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:20.148 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:20.148793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:20.200 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:20.200386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:30.314 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:30.314544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:30.328 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:30.328177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:40.438 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:40.438156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:40.448 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:40.448617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:50.551 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:50.551538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:23:50.562 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:23:50.562633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:00.666 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:00.666725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:00.679 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:00.679111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:10.778 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:10.778646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:10.875 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:10.875371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:20.974 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:20.974135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:20.986 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:20.986036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:31.150 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:31.150322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:31.165 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:31.165060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:41.270 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:41.270229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:41.281 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:41.281536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:51.385 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:51.385009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:24:51.396 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:24:51.396114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:01.485 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:01.485359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:01.499 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:01.499064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:11.706 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:11.706578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:11.717 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:11.717849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:21.781 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:21.780472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:21.792 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:21.792095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:31.991 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:31.991270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:32.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:32.004443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:42.312 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:42.311845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:42.324 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:42.324694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:52.410 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:52.410032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:25:52.441 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:25:52.441055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:02.615 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:02.614684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:02.628 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:02.628230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:12.726 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:12.726240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:12.738 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:12.738685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:22.797 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:22.796947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:22.809 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:22.809350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:32.912 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:32.912817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:32.924 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:32.924416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:43.022 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:43.022670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:43.034 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:43.034191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:53.165 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:53.165415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:26:53.182 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:26:53.181719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:03.299 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:03.298912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:03.310 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:03.310352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:13.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:13.414000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:13.426 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:13.427040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:23.499 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:23.499335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:23.562 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:23.562646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:33.661 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:33.660914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:33.674 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:33.674420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:43.737 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:43.737658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:43.800 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:43.800667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:53.902 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:53.902541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:27:53.914 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:27:53.914345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:04.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:04.004252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:04.018 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:04.018053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:14.081 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:14.081152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:14.092 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:14.092735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:24.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:24.158019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:24.233 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:24.233432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:34.305 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:34.305839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:34.315 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:34.315875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:44.379 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:44.379548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:44.394 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:44.394165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:54.449 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:54.449390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:28:54.459 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:28:54.459955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:04.565 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:04.564713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:04.576 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:04.576510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:14.736 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:14.735815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:14.805 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:14.748798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:24.849 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:24.849679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:24.861 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:24.861402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:35.009 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:35.009373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:35.018 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:35.019145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:45.126 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:45.126290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:45.141 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:45.141876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:55.280 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:55.279741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:29:55.295 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:29:55.295396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:05.467 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:05.467819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:05.478 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:05.479074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:15.573 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:15.573386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:15.585 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:15.585697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:25.688 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:25.688422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:25.701 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:25.701116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:35.768 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:35.768808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:35.781 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:35.781732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:45.868 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:45.868344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:45.880 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:45.880769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:55.938 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:55.938514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:30:55.973 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:30:55.973826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:06.038 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:06.039034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:06.051 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:06.051655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:16.131 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:16.131297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:16.143 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:16.143427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:26.374 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:26.374160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:26.388 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:26.388467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:36.487 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:36.487487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:36.500 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:36.500425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:46.562 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:46.562716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:46.574 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:46.574765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:56.660 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:56.660223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:31:56.670 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:31:56.670756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:06.788 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:06.788538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:06.803 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:06.802555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:16.917 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:16.916981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:16.928 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:16.928801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:27.017 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:27.016903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:27.032 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:27.031758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:37.096 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:37.096293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:37.106 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:37.106535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:47.175 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:47.175295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:47.186 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:47.186272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:57.253 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:57.253584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:32:57.267 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:32:57.267265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:07.321 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:07.321833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:07.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:07.333336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:17.454 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:17.454171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:17.464 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:17.464763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:27.608 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:27.608393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:27.621 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:27.620025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:37.673 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:37.672956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:37.687 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:37.687033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:43.886 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:43.886088 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T01:33:44.521 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:44.521245 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:33:44.513+0000 7fb63ca5c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:33:44.522 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T01:33:44.513+0000 7fb63ca5c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:33:44.522 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T01:33:44.513+0000 7fb63ca5c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:33:44.522 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:33:47.797 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:47.797612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:47.810 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:47.810289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:57.927 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:57.926749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:33:57.941 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:33:57.941266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:08.039 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:08.039351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:08.053 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:08.049468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:18.138 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:18.138302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:18.150 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:18.150172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:28.202 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:28.202345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:28.213 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:28.213551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:38.273 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:38.273873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:38.285 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:38.285472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:48.391 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:48.391178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:48.405 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:48.406350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:58.527 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:58.527763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:34:58.541 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:34:58.541443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:08.619 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:08.619548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:08.635 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:08.634952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:18.764 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:18.764321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:18.778 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:18.777989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:28.839 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:28.839707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:28.851 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:28.851587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:38.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:38.947155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:38.956 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:38.956854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:49.055 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:49.055023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:49.065 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:49.065932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:59.163 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:59.162563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:35:59.175 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:35:59.176140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:09.234 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:09.234540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:09.289 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:09.289412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:19.355 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:19.354871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:19.366 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:19.367050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:29.488 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:29.488308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:29.501 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:29.501441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:39.630 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:39.630525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:39.644 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:39.644333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:49.756 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:49.755701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:49.766 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:49.767144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:59.845 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:59.845446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:36:59.856 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:36:59.856496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:09.927 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:09.927099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:09.935 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:09.935968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:20.028 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:20.028866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:20.066 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:20.063401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:30.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:30.132461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:30.186 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:30.186323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:40.326 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:40.326110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:40.338 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:40.338791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:50.456 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:50.455957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:37:50.468 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:37:50.469066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:00.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:00.583596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:00.594 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:00.594820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:10.682 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:10.682534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:10.692 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:10.692459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:20.757 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:20.757136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:20.847 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:20.847551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:30.954 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:30.954565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:30.967 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:30.967168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:41.096 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:41.096042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:41.108 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:41.108626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:51.178 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:51.178246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:38:51.191 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:38:51.191278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:01.276 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:01.276313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:01.290 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:01.289050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:11.351 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:11.351945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:11.363 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:11.363360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:21.446 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:21.446007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:21.458 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:21.458940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:31.531 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:31.531058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:31.602 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:31.602190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:41.672 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:41.671830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:41.682 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:41.683143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:51.767 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:51.766355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:39:51.777 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:39:51.777766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:01.859 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:01.859543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:01.870 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:01.870855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:11.939 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:11.939414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:11.951 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:11.951133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:22.013 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:22.013785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:22.025 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:22.025793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:32.099 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:32.099833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:32.113 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:32.113099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:42.171 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:42.171153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:42.182 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:42.182781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:52.249 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:52.249287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:40:52.311 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:40:52.311195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:02.374 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:02.374654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:02.385 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:02.385930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:12.445 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:12.445805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:12.458 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:12.458651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:22.540 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:22.540140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:22.552 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:22.553183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:32.625 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:32.625021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:32.682 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:32.680778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:42.753 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:42.753255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:42.765 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:42.766053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:52.829 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:52.829569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:41:52.839 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:41:52.839916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:03.009 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:03.009351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:03.020 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:03.019732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:13.101 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:13.101505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:13.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:13.112951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:23.177 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:23.177413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:23.190 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:23.189844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:33.262 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:33.262204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:33.316 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:33.316895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:43.418 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:43.418233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:43.431 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:43.432057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:53.496 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:53.495457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:42:53.508 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:42:53.508060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:03.650 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:03.650070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:03.661 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:03.661464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:13.740 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:13.740348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:13.757 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:13.757786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:23.833 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:23.833561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:23.845 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:23.845288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:33.912 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:33.912096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:33.924 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:33.924533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:44.042 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:44.042170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:44.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:44.054810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:54.113 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:54.113561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:43:54.161 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:43:54.161178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:04.235 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:04.235354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:04.284 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:04.280031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:14.447 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:14.447129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:14.457 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:14.458202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:24.609 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:24.609575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:24.619 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:24.619557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:34.725 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:34.724878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:34.740 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:34.740900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:44.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:44.855486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:44.867 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:44.867757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:54.972 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:54.972049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:44:54.983 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:44:54.983493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:05.036 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:05.037025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:05.089 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:05.089514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:15.151 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:15.151751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:15.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:15.167449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:25.274 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:25.274287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:25.285 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:25.285876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:35.370 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:35.370128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:35.387 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:35.387478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:45.495 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:45.495379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:45.507 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:45.507104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:55.583 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:55.583643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:45:55.594 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:45:55.594741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:05.674 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:05.674808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:05.686 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:05.686983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:15.757 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:15.757561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:15.766 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:15.766949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:25.860 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:25.859991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:25.873 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:25.873571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:35.925 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:35.925931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:35.936 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:35.936788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:46.021 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:46.021019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:46.035 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:46.034399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:56.153 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:56.153015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:46:56.164 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:46:56.164991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:06.281 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:06.281680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:06.302 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:06.302646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:16.434 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:16.434829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:16.446 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:16.446663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:26.541 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:26.541196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:26.554 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:26.554974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:36.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:36.663855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:36.673 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:36.673906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:46.782 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:46.782846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:46.794 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:46.794166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:56.894 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:56.894096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:47:56.905 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:47:56.905855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:07.005 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:07.005768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:07.017 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:07.018063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:17.114 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:17.114866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:17.128 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:17.128279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:27.271 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:27.271307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:27.282 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:27.282171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:37.375 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:37.375121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:37.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:37.415264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:47.484 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:47.484472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:47.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:47.495236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:57.594 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:57.594490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:48:57.605 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:48:57.605683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:07.713 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:07.712680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:07.729 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:07.729243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:17.807 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:17.807608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:17.818 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:17.819035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:27.889 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:27.889471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:27.902 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:27.902515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:37.995 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:37.995323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:38.015 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:38.015328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:48.140 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:48.140727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:48.153 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:48.153985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:58.217 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:58.218027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:49:58.232 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:49:58.232351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:08.335 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:08.335496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:08.350 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:08.350299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:18.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:18.455879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:18.467 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:18.467403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:24.532 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:24.532131 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T01:50:25.166 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:25.164614 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T01:50:25.152+0000 7fdc9cd6c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T01:50:25.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T01:50:25.152+0000 7fdc9cd6c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T01:50:25.168 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T01:50:25.152+0000 7fdc9cd6c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T01:50:25.168 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T01:50:28.545 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:28.545994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:28.563 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:28.562818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:38.640 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:38.640240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:38.706 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:38.706001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:48.760 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:48.760942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:48.772 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:48.772705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:58.853 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:58.853412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:50:58.864 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:50:58.865053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:08.980 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:08.980866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:08.993 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:08.993675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:19.060 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:19.060611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:19.073 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:19.074052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:29.180 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:29.180248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:29.192 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:29.193216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:39.262 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:39.262183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:39.319 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:39.319280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:49.394 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:49.394441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:49.407 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:49.407276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:59.477 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:59.477291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:51:59.557 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:51:59.557729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:09.649 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:09.649478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:09.662 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:09.662693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:19.733 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:19.733383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:19.744 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:19.745024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:29.863 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:29.863936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:29.875 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:29.875761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:39.944 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:39.944274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:39.956 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:39.956480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:50.026 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:50.026302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:52:50.085 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:52:50.085333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:00.176 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:00.176370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:00.188 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:00.189236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:10.266 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:10.266218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:10.278 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:10.278911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:20.339 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:20.338950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:20.350 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:20.350813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:30.494 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:30.494711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:30.505 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:30.505255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:40.608 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:40.608250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:40.623 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:40.623871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:50.675 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:50.674827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:53:50.739 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:53:50.739403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:00.915 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:00.915852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:00.925 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:00.925178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:11.019 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:11.018979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:11.031 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:11.032113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:21.135 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:21.135614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:21.150 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:21.151013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:31.217 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:31.217696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:31.230 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:31.231125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:41.357 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:41.357175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:41.368 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:41.368654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:51.454 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:51.453804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:54:51.469 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:54:51.465757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:01.564 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:01.564189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:01.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:01.581526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:11.647 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:11.647314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:11.658 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:11.658714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:21.766 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:21.767033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:21.779 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:21.779444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:31.883 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:31.882802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:31.895 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:31.894888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:42.004 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:42.004152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:42.018 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:42.018665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:52.092 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:52.092972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:55:52.145 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:55:52.145284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:02.200 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:02.200772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:02.211 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:02.212106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:12.320 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:12.320673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:12.332 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:12.333049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:22.439 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:22.438868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:22.449 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:22.450107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:32.521 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:32.521331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:32.533 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:32.533750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:42.650 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:42.650029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:42.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:42.663394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:52.763 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:52.762955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:56:52.776 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:56:52.776862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:02.845 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:02.838909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:02.851 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:02.851273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:12.922 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:12.921640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:12.933 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:12.934108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:23.018 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:23.018759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:23.032 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:23.032592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:33.098 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:33.096074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:33.107 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:33.107443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:43.232 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:43.232459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:43.247 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:43.247478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:53.365 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:53.365941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:57:53.376 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:57:53.376951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:03.454 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:03.454845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:03.465 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:03.466191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:13.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:13.588746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:13.603 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:13.603337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:23.670 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:23.670079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:23.681 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:23.681856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:33.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:33.771631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:33.785 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:33.785389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:43.844 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:43.844519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:43.855 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:43.856073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:53.976 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:53.976803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:58:53.987 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:58:53.987764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:04.124 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:04.125018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:04.139 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:04.139825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:14.233 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:14.233823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:14.271 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:14.267146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:24.363 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:24.364005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:24.375 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:24.376038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:34.477 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:34.477104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:34.487 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:34.487853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:44.603 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:44.603353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:44.614 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:44.614249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:54.704 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:54.704440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T01:59:54.715 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 01:59:54.716048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:04.837 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:04.836779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:04.851 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:04.851479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:14.920 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:14.919910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:14.930 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:14.930619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:25.025 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:25.025341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:25.040 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:25.040416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:35.100 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:35.100596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:35.113 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:35.113066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:45.235 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:45.235216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:45.248 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:45.249178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:55.314 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:55.314776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:00:55.325 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:00:55.326094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:05.393 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:05.393222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:05.403 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:05.403311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:15.469 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:15.469855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:15.507 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:15.508124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:25.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:25.600740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:25.611 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:25.612215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:35.728 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:35.728998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:35.743 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:35.743212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:45.857 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:45.857287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:45.872 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:45.871469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:55.961 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:55.961757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:01:55.972 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:01:55.972884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:06.081 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:06.081710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:06.094 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:06.094402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:16.168 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:16.168599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:16.178 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:16.179087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:26.289 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:26.289939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:26.301 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:26.301586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:36.403 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:36.403424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:36.419 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:36.420019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:46.589 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:46.589297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:46.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:46.600784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:56.667 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:56.667112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:02:56.679 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:02:56.679391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:06.772 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:06.772661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:06.784 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:06.784438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:16.859 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:16.859954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:16.871 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:16.871700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:26.981 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:26.981506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:27.025 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:27.008128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:37.116 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:37.116068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:37.175 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:37.175057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:47.273 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:47.273964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:47.288 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:47.288866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:57.350 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:57.350320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:03:57.360 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:03:57.360898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:07.423 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:07.424051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:07.508 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:07.508746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:17.579 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:17.579245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:17.591 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:17.591357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:27.675 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:27.675816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:27.694 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:27.694846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:37.884 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:37.884018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:37.894 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:37.894301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:48.100 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:48.096061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:48.111 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:48.111627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:58.175 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:58.175623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:04:58.237 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:04:58.237181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:08.292 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:08.292183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:08.303 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:08.303515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:18.366 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:18.367138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:18.379 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:18.379551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:28.480 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:28.480094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:28.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:28.491737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:38.549 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:38.548428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:38.560 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:38.561093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:48.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:48.639244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:48.648 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:48.648956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:58.708 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:58.708501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:05:58.721 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:05:58.721164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:08.819 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:08.819293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:08.829 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:08.829716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:18.893 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:18.893981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:18.947 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:18.946741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:29.040 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:29.040208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:29.053 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:29.053736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:39.149 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:39.148975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:39.161 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:39.161315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:49.221 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:49.221069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:49.233 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:49.233296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:59.289 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:59.286898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:06:59.298 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:06:59.298365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:05.174 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:05.174514 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T02:07:05.818 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:05.818527 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:07:05.807+0000 7f19de754380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:07:05.819 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T02:07:05.807+0000 7f19de754380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:07:05.819 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T02:07:05.807+0000 7f19de754380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:07:05.819 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:07:09.403 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:09.403941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:09.414 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:09.414185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:19.521 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:19.521769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:19.534 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:19.534647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:29.639 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:29.639487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:29.652 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:29.652966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:39.712 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:39.712613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:39.724 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:39.724465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:49.818 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:49.818008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:49.829 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:49.829842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:59.904 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:59.905008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:07:59.916 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:07:59.916858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:09.984 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:09.984421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:09.997 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:09.997535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:20.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:20.052863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:20.064 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:20.064529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:30.190 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:30.190362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:30.204 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:30.204415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:40.345 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:40.345238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:40.357 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:40.357341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:50.495 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:50.495860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:08:50.509 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:08:50.508716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:00.574 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:00.574645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:00.587 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:00.587538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:10.646 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:10.646106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:10.656 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:10.657223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:20.740 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:20.740135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:20.803 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:20.803756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:30.870 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:30.870716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:30.883 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:30.882360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:40.970 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:40.970586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:40.982 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:40.982752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:51.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:51.054559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:09:51.066 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:09:51.067104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:01.187 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:01.188020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:01.201 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:01.202134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:11.303 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:11.303381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:11.317 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:11.317819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:21.434 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:21.433929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:21.445 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:21.445497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:31.549 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:31.549066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:31.564 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:31.563936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:41.718 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:41.718429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:41.732 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:41.732307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:51.900 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:51.901060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:10:51.914 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:10:51.913856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:01.978 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:01.978358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:02.030 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:02.030264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:12.088 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:12.088090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:12.099 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:12.099893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:22.195 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:22.195892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:22.209 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:22.209783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:32.323 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:32.323255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:32.335 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:32.335359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:42.424 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:42.424769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:42.438 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:42.437915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:52.539 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:52.539538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:11:52.551 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:11:52.551694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:02.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:02.663463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:02.676 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:02.676514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:12.784 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:12.784860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:12.795 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:12.796154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:22.894 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:22.894070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:22.904 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:22.904440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:32.962 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:32.962584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:32.977 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:32.974785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:43.039 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:43.038962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:43.050 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:43.050513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:53.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:53.112948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:12:53.126 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:12:53.126862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:03.190 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:03.190831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:03.202 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:03.202888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:13.303 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:13.303095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:13.316 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:13.316920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:23.409 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:23.409026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:23.424 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:23.424571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:33.523 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:33.523389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:33.533 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:33.533360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:43.650 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:43.649809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:43.663 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:43.662944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:53.765 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:53.764673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:13:53.796 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:13:53.796179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:03.854 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:03.853891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:03.865 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:03.865847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:13.923 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:13.923255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:13.936 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:13.936435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:24.072 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:24.072606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:24.084 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:24.085279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:34.209 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:34.209230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:34.219 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:34.219492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:44.308 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:44.308796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:44.325 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:44.320649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:54.580 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:54.580471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:14:54.596 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:14:54.596774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:04.661 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:04.661224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:04.674 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:04.674121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:14.758 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:14.758182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:14.771 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:14.771683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:24.846 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:24.846359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:24.859 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:24.860195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:34.960 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:34.960606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:34.973 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:34.972274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:45.052 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:45.052504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:45.065 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:45.065781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:55.205 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:55.205535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:15:55.216 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:15:55.216634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:05.274 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:05.274532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:05.286 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:05.286287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:15.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:15.378762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:15.391 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:15.390934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:25.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:25.491949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:25.503 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:25.503536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:35.608 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:35.608810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:35.618 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:35.618886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:45.724 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:45.724710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:45.739 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:45.739875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:55.870 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:55.870388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:16:55.881 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:16:55.882051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:05.977 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:05.977260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:05.997 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:05.997081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:16.057 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:16.056988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:16.070 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:16.070171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:26.147 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:26.146992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:26.205 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:26.205656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:36.272 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:36.272225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:36.281 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:36.282106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:46.352 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:46.352391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:46.369 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:46.369294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:56.486 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:56.486155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:17:56.497 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:17:56.497555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:06.607 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:06.606776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:06.621 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:06.621330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:16.734 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:16.733832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:16.747 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:16.747362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:26.813 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:26.812953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:26.825 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:26.826102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:36.956 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:36.956183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:36.967 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:36.967956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:47.049 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:47.049064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:47.061 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:47.061589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:57.156 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:57.156864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:18:57.169 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:18:57.170251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:07.265 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:07.265342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:07.279 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:07.279604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:17.364 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:17.364281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:17.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:17.378052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:27.437 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:27.437528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:27.448 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:27.448469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:37.560 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:37.560505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:37.572 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:37.572385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:47.685 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:47.685067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:47.698 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:47.698970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:57.802 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:57.802862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:19:57.844 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:19:57.844275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:07.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:07.946841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:07.956 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:07.956508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:18.070 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:18.070710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:18.080 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:18.081245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:28.179 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:28.179331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:28.192 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:28.192625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:38.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:38.259325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:38.271 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:38.271797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:48.378 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:48.378138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:48.390 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:48.391012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:58.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:58.491824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:20:58.507 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:20:58.507004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:08.573 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:08.573566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:08.587 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:08.587347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:18.682 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:18.682700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:18.719 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:18.719237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:28.798 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:28.798216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:28.812 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:28.812584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:38.919 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:38.919932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:38.930 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:38.931449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:48.993 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:48.993638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:49.003 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:49.003201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:59.129 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:59.129647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:21:59.142 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:21:59.142619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:09.210 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:09.210571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:09.221 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:09.222121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:19.294 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:19.294851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:19.306 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:19.306896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:29.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:29.415515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:29.428 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:29.429142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:39.501 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:39.501294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:39.576 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:39.576380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:49.644 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:49.644193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:49.688 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:49.688034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:59.763 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:59.763128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:22:59.775 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:22:59.775447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:09.889 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:09.889414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:09.902 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:09.902794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:20.011 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:20.011873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:20.024 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:20.024315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:30.106 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:30.106646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:30.162 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:30.162290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:40.280 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:40.280719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:40.297 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:40.297804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:45.825 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:45.825659 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T02:23:46.520 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:46.519610 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:23:46.510+0000 7f82b26d5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:23:46.520 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T02:23:46.510+0000 7f82b26d5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:23:46.520 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T02:23:46.510+0000 7f82b26d5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:23:46.520 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:23:50.429 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:50.429476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:23:50.440 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:23:50.440962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:00.550 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:00.550517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:00.563 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:00.562558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:10.617 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:10.617657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:10.629 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:10.629693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:20.697 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:20.698055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:20.710 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:20.711285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:30.779 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:30.779525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:30.794 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:30.794280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:40.862 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:40.862148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:40.912 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:40.912406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:51.008 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:51.008657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:24:51.022 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:24:51.022860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:01.135 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:01.135170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:01.151 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:01.151759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:11.249 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:11.249214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:11.260 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:11.261214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:21.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:21.415963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:21.428 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:21.428674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:31.595 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:31.596105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:31.610 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:31.610411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:41.714 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:41.714028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:41.723 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:41.723291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:51.853 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:51.847869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:25:51.890 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:25:51.890127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:01.963 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:01.964066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:01.975 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:01.975891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:12.119 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:12.119591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:12.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:12.132297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:22.236 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:22.236903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:22.250 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:22.250452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:32.311 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:32.311794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:32.398 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:32.398103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:42.527 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:42.527278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:42.539 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:42.539719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:52.629 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:52.629268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:26:52.642 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:26:52.642865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:02.863 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:02.862346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:02.871 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:02.871627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:12.946 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:12.946352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:12.955 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:12.956225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:23.080 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:23.080550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:23.092 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:23.093184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:33.179 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:33.179520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:33.190 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:33.191172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:43.316 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:43.316820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:43.330 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:43.330871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:53.405 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:53.405885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:27:53.418 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:27:53.418323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:03.473 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:03.473830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:03.484 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:03.484433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:13.579 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:13.579474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:13.593 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:13.593285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:23.681 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:23.681037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:23.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:23.695509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:33.816 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:33.816623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:33.829 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:33.829410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:43.940 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:43.940600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:43.952 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:43.952725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:54.054 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:54.054889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:28:54.068 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:28:54.068107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:04.135 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:04.135689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:04.145 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:04.146079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:14.253 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:14.252898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:14.267 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:14.267349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:24.366 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:24.366434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:24.376 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:24.376892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:34.449 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:34.449952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:34.490 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:34.490428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:44.581 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:44.581959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:44.594 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:44.594304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:54.696 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:54.696729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:29:54.709 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:29:54.710102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:04.839 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:04.839418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:04.851 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:04.851308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:14.923 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:14.923531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:14.932 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:14.933061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:25.042 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:25.042924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:25.055 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:25.055496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:35.182 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:35.182083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:35.196 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:35.196517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:45.265 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:45.265800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:45.276 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:45.276594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:55.387 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:55.387299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:30:55.399 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:30:55.399631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:05.480 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:05.480835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:05.492 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:05.492509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:15.589 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:15.589377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:15.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:15.600750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:25.703 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:25.704002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:25.717 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:25.717560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:35.783 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:35.784042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:35.795 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:35.796185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:45.900 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:45.900194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:45.954 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:45.954299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:56.066 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:56.066161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:31:56.078 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:31:56.079099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:06.138 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:06.138516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:06.148 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:06.148437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:16.249 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:16.249865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:16.261 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:16.262012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:26.321 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:26.321857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:26.332 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:26.332438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:36.395 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:36.395567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:36.448 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:36.448702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:46.516 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:46.516872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:46.527 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:46.527899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:56.631 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:56.630860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:32:56.643 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:32:56.643688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:06.805 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:06.804692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:06.816 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:06.816171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:16.887 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:16.887886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:16.902 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:16.903127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:26.970 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:26.969915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:26.981 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:26.981774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:37.048 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:37.047899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:37.060 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:37.060298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:47.162 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:47.157263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:47.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:47.168128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:57.288 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:57.288891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:33:57.299 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:33:57.299671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:07.400 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:07.399762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:07.415 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:07.415468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:17.488 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:17.488214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:17.549 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:17.497733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:27.578 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:27.578271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:27.591 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:27.591535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:37.701 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:37.701086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:37.713 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:37.713378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:47.781 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:47.780619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:47.798 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:47.798638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:57.870 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:57.870647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:34:57.882 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:34:57.882977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:07.986 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:07.986499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:07.997 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:07.997617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:18.145 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:18.145715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:18.159 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:18.158846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:28.221 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:28.221395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:28.272 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:28.272847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:38.395 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:38.395677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:38.409 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:38.410003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:48.488 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:48.488265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:48.500 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:48.500761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:58.596 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:58.595767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:35:58.608 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:35:58.608260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:08.709 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:08.709046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:08.723 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:08.723302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:18.822 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:18.822646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:18.836 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:18.836300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:28.913 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:28.913003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:28.930 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:28.930767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:39.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:39.023753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:39.033 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:39.033145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:49.140 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:49.140206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:49.151 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:49.151556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:59.228 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:59.228305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:36:59.239 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:36:59.239993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:09.321 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:09.321679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:09.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:09.333000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:19.409 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:19.409796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:19.423 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:19.423549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:29.485 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:29.484910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:29.498 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:29.498931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:39.570 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:39.569785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:39.584 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:39.583271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:49.685 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:49.685447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:49.698 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:49.698807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:59.761 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:59.760882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:37:59.772 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:37:59.772585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:09.874 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:09.873719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:09.887 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:09.886843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:19.978 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:19.977786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:19.991 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:19.991394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:30.087 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:30.087632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:30.102 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:30.102087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:40.189 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:40.189765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:40.205 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:40.205315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:50.271 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:50.271741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:38:50.285 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:38:50.285142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:00.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:00.353494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:00.364 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:00.364545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:10.433 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:10.432899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:10.442 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:10.441829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:20.523 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:20.523126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:20.631 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:20.534421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:30.613 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:30.613462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:30.625 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:30.624865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:40.740 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:40.740210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:40.754 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:40.754577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:50.849 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:50.849369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:39:50.860 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:39:50.860776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:00.925 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:00.925730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:00.939 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:00.938986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:11.047 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:11.046782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:11.059 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:11.058958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:21.125 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:21.125065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:21.137 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:21.137274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:26.530 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:26.529635 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T02:40:27.166 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:27.166158 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:40:27.157+0000 7f18df62c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:40:27.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T02:40:27.157+0000 7f18df62c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:40:27.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T02:40:27.157+0000 7f18df62c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:40:27.167 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:40:31.223 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:31.223254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:31.235 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:31.235208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:41.362 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:41.362293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:41.380 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:41.381004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:51.443 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:51.442501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:40:51.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:40:51.455853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:01.559 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:01.559306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:01.576 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:01.576964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:11.681 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:11.680974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:11.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:11.694971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:21.780 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:21.780184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:21.797 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:21.796942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:31.867 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:31.867387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:31.879 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:31.879909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:41.950 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:41.949837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:42.013 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:42.012436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:52.092 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:52.091857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:41:52.141 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:41:52.141462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:02.232 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:02.232062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:02.244 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:02.244495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:12.335 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:12.334751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:12.347 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:12.347250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:22.421 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:22.420853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:22.440 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:22.440790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:32.601 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:32.600970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:32.611 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:32.610944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:42.754 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:42.754027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:42.814 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:42.814363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:52.906 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:52.900813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:42:52.916 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:42:52.916183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:02.993 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:02.993012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:03.028 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:03.028102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:13.112 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:13.112396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:13.123 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:13.123245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:23.289 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:23.289421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:23.306 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:23.305776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:33.471 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:33.471197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:33.481 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:33.481262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:43.679 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:43.678748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:43.690 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:43.690375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:53.792 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:53.791812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:43:53.804 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:43:53.804569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:03.859 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:03.856778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:03.909 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:03.905893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:14.046 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:14.046332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:14.058 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:14.057965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:24.186 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:24.185859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:24.199 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:24.198837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:34.300 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:34.299805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:34.313 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:34.311536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:44.406 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:44.406122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:44.419 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:44.418368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:54.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:54.491337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:44:54.502 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:44:54.502844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:04.587 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:04.587265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:04.600 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:04.599782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:14.695 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:14.695507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:14.708 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:14.707933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:24.820 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:24.819746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:24.830 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:24.830634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:34.937 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:34.937496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:34.950 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:34.950457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:45.020 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:45.019596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:45.031 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:45.031677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:55.114 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:55.113949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:45:55.193 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:45:55.127107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:05.211 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:05.210928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:05.222 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:05.221908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:15.317 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:15.316851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:15.329 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:15.329439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:25.416 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:25.416259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:25.427 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:25.427597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:35.527 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:35.527427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:35.539 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:35.538826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:45.603 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:45.603468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:45.649 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:45.648686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:55.722 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:55.722082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:46:55.735 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:46:55.735439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:05.829 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:05.829546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:05.841 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:05.841495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:15.952 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:15.951641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:15.968 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:15.968426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:26.115 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:26.115112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:26.126 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:26.125105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:36.193 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:36.193371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:36.247 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:36.247450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:46.315 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:46.315041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:46.327 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:46.326797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:56.395 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:56.394727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:47:56.408 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:47:56.408628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:06.499 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:06.499301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:06.511 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:06.511181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:16.605 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:16.604436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:16.618 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:16.617320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:26.708 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:26.707480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:26.720 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:26.719429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:36.818 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:36.817563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:36.830 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:36.829949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:46.900 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:46.899901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:46.914 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:46.914193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:57.022 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:57.021997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:48:57.036 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:48:57.036309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:07.105 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:07.104953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:07.117 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:07.117372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:17.222 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:17.221884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:17.235 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:17.234566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:27.288 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:27.286425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:27.295 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:27.295732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:37.393 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:37.392815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:37.407 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:37.406844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:47.473 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:47.473512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:47.485 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:47.484932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:57.550 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:57.549794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:49:57.562 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:49:57.561705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:07.621 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:07.621217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:07.634 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:07.633811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:17.760 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:17.760300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:17.773 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:17.773148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:27.837 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:27.836890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:27.850 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:27.850277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:37.952 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:37.952474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:37.961 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:37.961399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:48.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:48.023073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:48.035 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:48.034840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:58.145 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:58.144804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:50:58.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:50:58.156801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:08.259 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:08.259508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:08.272 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:08.271940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:18.365 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:18.364017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:18.375 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:18.374683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:28.482 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:28.481010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:28.495 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:28.494571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:38.560 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:38.559670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:38.573 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:38.572540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:48.676 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:48.675514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:48.689 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:48.687233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:58.785 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:58.784354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:51:58.798 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:51:58.797460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:08.908 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:08.908383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:08.921 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:08.920542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:19.015 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:19.014400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:19.072 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:19.071816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:29.195 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:29.194733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:29.207 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:29.206689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:39.321 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:39.320997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:39.333 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:39.333150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:49.420 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:49.419574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:49.433 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:49.432909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:59.542 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:59.541530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:52:59.557 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:52:59.556713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:09.630 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:09.629914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:09.641 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:09.641133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:19.709 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:19.709045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:19.767 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:19.767210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:29.830 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:29.830286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:29.843 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:29.842184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:39.937 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:39.936707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:39.950 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:39.949831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:50.076 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:50.075621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:53:50.088 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:53:50.088222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:00.167 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:00.166813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:00.177 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:00.177436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:10.292 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:10.292156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:10.307 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:10.306323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:20.408 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:20.407979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:20.421 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:20.420777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:30.489 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:30.488505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:30.502 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:30.501790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:40.624 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:40.624054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:40.638 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:40.637602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:50.728 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:50.727187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:54:50.741 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:54:50.740389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:00.846 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:00.845346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:00.859 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:00.859373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:10.985 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:10.985088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:10.999 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:10.999109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:21.072 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:21.071725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:21.083 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:21.082923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:31.146 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:31.145973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:31.195 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:31.194874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:41.353 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:41.352222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:41.364 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:41.363855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:51.448 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:51.447483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:55:51.461 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:55:51.460759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:01.575 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:01.574648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:01.588 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:01.588150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:11.691 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:11.691211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:11.702 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:11.702116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:21.812 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:21.803328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:22.014 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:22.013554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:32.198 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:32.197226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:32.210 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:32.208904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:42.306 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:42.305517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:42.318 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:42.317973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:52.407 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:52.406791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:56:52.421 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:56:52.421042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:02.491 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:02.491330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:02.504 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:02.503692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:07.174 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:07.174263 I | op-mon: parsing mon endpoints: a=10.99.149.173:6789,b=10.100.13.26:6789,c=10.97.212.176:6789 2022-03-09T02:57:07.829 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:07.828434 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-09T02:57:07.816+0000 7f48e1db5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-09T02:57:07.829 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T02:57:07.816+0000 7f48e1db5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-09T02:57:07.830 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09T02:57:07.816+0000 7f48e1db5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-09T02:57:07.830 INFO:tasks.rook.operator.smithi107.stdout:couldn't init storage provider. : exit status 5 2022-03-09T02:57:12.569 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:12.568964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:12.620 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:12.620338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:22.712 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:22.711801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:22.723 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:22.723293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:32.827 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:32.827416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:32.841 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:32.840917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:42.955 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:42.954508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:42.965 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:42.963673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:53.031 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:53.030883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:57:53.040 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:57:53.040618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:03.141 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:03.140943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:03.154 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:03.153707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:13.219 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:13.218178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:13.230 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:13.230446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:23.346 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:23.344904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:23.357 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:23.356832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:33.420 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:33.419509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:33.433 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:33.433489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:43.495 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:43.494669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:43.509 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:43.509389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:53.571 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:53.570444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:58:53.582 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:58:53.581526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:03.646 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:03.642328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:03.657 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:03.656814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:13.761 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:13.761283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:13.803 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:13.773998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:23.891 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:23.890258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:23.903 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:23.903295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:34.023 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:34.023385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:34.037 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:34.036630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:44.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:44.131718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:44.144 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:44.144314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:54.261 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:54.261236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T02:59:54.274 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 02:59:54.273961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:04.346 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:04.345481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:04.359 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:04.358962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:14.455 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:14.455273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:14.466 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:14.465763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:24.533 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:24.532606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:24.545 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:24.544936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:34.650 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:34.649979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:34.662 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:34.661819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:44.775 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:44.774640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:44.788 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:44.788120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:54.913 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:54.913275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:00:54.930 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:00:54.929957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:04.999 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:04.999239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:05.013 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:05.012972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:15.074 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:15.074114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:15.086 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:15.086335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:25.145 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:25.145164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:25.157 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:25.156888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:35.255 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:35.254691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:35.267 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:35.266512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:45.329 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:45.328939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:45.343 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:45.342710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:55.442 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:55.441281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:01:55.468 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:01:55.468168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:05.550 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:05.549841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:05.562 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:05.561567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:15.653 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:15.653188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:15.666 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:15.665904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:25.774 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:25.773550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:25.785 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:25.785154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:35.848 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:35.847671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:35.860 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:35.860509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:45.967 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:45.966681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:45.978 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:45.977918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:56.056 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:56.055874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:02:56.067 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:02:56.066870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:06.161 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:06.160945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:06.173 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:06.172728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:16.281 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:16.280890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:16.297 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:16.296557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:26.403 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:26.402236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:26.416 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:26.416250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:36.596 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:36.595809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:36.608 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:36.608190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:46.677 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:46.677349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:46.724 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:46.723634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:56.830 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:56.829631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:03:56.838 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:03:56.838356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:06.895 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:06.895259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:06.942 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:06.941613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:17.014 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:17.014104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:17.028 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:17.028380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:27.132 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:27.131675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:27.145 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:27.144630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:37.282 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:37.281836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:37.293 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:37.292580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:47.423 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:47.422606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:47.434 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:47.434392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:57.605 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:57.604652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:04:57.618 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:04:57.617634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:05:07.744 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:05:07.743965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T03:05:07.754 INFO:tasks.rook.operator.smithi107.stdout:2022-03-09 03:05:07.754073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-09T05:25:34.351 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-09T05:25:34.380 DEBUG:teuthology.task.console_log:Killing console logger for smithi107 2022-03-09T05:25:34.381 DEBUG:teuthology.task.console_log:Killing console logger for smithi155 2022-03-09T05:25:34.382 DEBUG:teuthology.task.console_log:Killing console logger for smithi187 2022-03-09T05:25:34.382 DEBUG:teuthology.exit:Finished running handlers