2022-03-08T04:12:07.266 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-08T04:12:07.268 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} --name yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725871 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725871/orig.config.yaml 2022-03-08T04:12:07.303 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-08T04:12:07.365 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725871 branch: wip-yuri5-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} email: ceph-qa@ceph.io first_in_suite: false job_id: '6725871' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2022-03-07-0958 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri5-testing-2022-03-07-0958 sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri5-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 targets: smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxzYZRYHPySRhFII4EpKaoEHYmF5bj70CDgCu1BUXe6uODU6/kUjApoRPc+wuGAzE++3W1ztlaHF1IaFlCNZxctqcuSrK2eA/XLesCCna8hUANcKvn/yHz+Pucq4Dw3sYRBuWb7L1WZRnLXbI7z4aaMhHo2o5edPQjylfI0c3DTDtZEPW1m80Nt7PAAwPalbMW1a15MwzmH/OPaMbzQReKiYXG69c0Opck2XLku67ScvvhChsCEbz+Pv2sfSGKrsA7F1sUWfFVOykDZbu8Q1bZ4M5hiLLQgBIjlmH1OQv1HEUN4r0DxB4HuG/YTeD9AX6SeUgVXc+UtS0CHNDSnDKB8w2I7YQKp86yKFtEqD7kkuC3rfsA8hNgUlORntdVKIUPtm3pn6fItJb+n+JpWWjmL4LV8gSi/22ywSzJy3NCtJoDdgxpO5uiodw/TfijpQr5mtVK/WPRT0ohoLd5SDlPuw9IZ1smQeuIDZmjqjYKcUt89FR737dYRj2W7RC178= smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRfoUSOTSQ71WBGVtFlMjtg10hCvGFwLnZNupxG6XFnHzcJRjpNGhV0x95I3UATlE+rtA3SLYeos55X/NIeSx6FX9PIjlFCQIYio+s5hOpH95SeiPalOGclkVjYf4Fvp06a7C5IHcZXdjABVvIESucV9PQy+NdcFSFnjpzNUtVpoensXz90ry/UVcYWhm1LeYoEc0K9MSDbzczTfPG32xbhy1j0qFO/5iEMUeygBgwMR0YkmZsFSj3+U/vK31YGG6+e3MnKhrna6N0SsFIER7E+fNrCALHdk4TrZyK8dy8Ki11q/LgAY8kg0+WF86oBxQHoYn9r8KRC+5fqoVcWIoG/CCfVbciAJGmfNdODwJgXszYtQT1Y2XEPqPNaGrhPnoVbl2hyRMo6iOnAZOj1t3ZN5O/IHG25bua8fXV/pEyL3QCVSF+PNAkKui6pP1/xQY3Cp+6mfhWkBnOun0vi7F04AhMWCxPYYsPegIINoiA+JRE3YrpXksW0s4UdWgz/R8= smithi105.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC39t8Zn8/cB1IkBLkYzM6SulaKaH4OETXjA+Mo2ppc9E3UxrMDgI5t3tvDf1UAbSH/6JTJMTWDQJfMYLeHeFY0ZXuf3nqKlSnUvZLsoAwsaRvdb8z29pcLe/jbR4X9LJunO3mhbdJYr5kqmpbFULmyGRu8L7n46fnxq/ndjURFIFhZ4pr1K94wpy6vvwCKjHpoYioAc+veLXhVkXp1vG4mzTyQiet67zmMVZLl7zp8ZD04nSsCcm5fYQtr5tofwWh7VZNx2IAbe4zY3tz+yCKYnCpqV7BWm24ViJj9E1tczErifxBLnk6BNaUu8JrL/nSTa/Y8o1ParCJ9hXUM5l6IVgiBee/bv8GEHrGjgSEezguPi4o1XVzbhU2kRUTNV+M8YRkj1VhYpoI7tfp33zzyRziLZP9/as1l3D20YsZLYIycPPLURKB4KmSdghBMenELaSiRIV7AFSo8wB3A0hFhsstBL5kXnN+JA+ddtAJSXTegfrqsX8qdNLrtXmM7DVM= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-07_22:09:03 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-08T04:12:07.366 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa; will attempt to use it 2022-03-08T04:12:07.366 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks 2022-03-08T04:12:07.367 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-08T04:12:07.369 INFO:teuthology.task.internal:Checking packages... 2022-03-08T04:12:07.390 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7a44e9ebca556862e9704be295a07719d7db47b3' 2022-03-08T04:12:07.390 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-08T04:12:07.391 INFO:teuthology.packaging:ref: None 2022-03-08T04:12:07.391 INFO:teuthology.packaging:tag: None 2022-03-08T04:12:07.391 INFO:teuthology.packaging:branch: wip-yuri5-testing-2022-03-07-0958 2022-03-08T04:12:07.392 INFO:teuthology.packaging:sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T04:12:07.392 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2022-03-07-0958 2022-03-08T04:12:07.550 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11025-g7a44e9eb-1focal 2022-03-08T04:12:07.551 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-08T04:12:07.563 INFO:teuthology.task.internal:no buildpackages task found 2022-03-08T04:12:07.564 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-08T04:12:07.676 INFO:teuthology.task.internal:Saving configuration 2022-03-08T04:12:07.688 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-08T04:12:07.703 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-08T04:12:07.726 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725871', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 04:06:47.254248', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHvlowCyL2OAyYh77FBYfxhH+5zs0VCID40RCdbMJSBo'} 2022-03-08T04:12:07.749 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725871', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 04:06:47.252922', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINKzWHJVi8cltNyiXBDto5buDiz37RNcdpe2Cz4ydwjd'} 2022-03-08T04:12:07.772 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-07_22:09:03-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6725871', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-08 04:06:47.255285', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICQs30HLgmARi2hz+n1/incCoxuu3LabnbS3hDFjwys2'} 2022-03-08T04:12:07.773 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-08T04:12:07.789 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-08T04:12:07.789 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-08T04:12:07.789 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-08T04:12:07.790 INFO:teuthology.run_tasks:Running task console_log... 2022-03-08T04:12:07.905 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fce0e5859d8>, signals=[15]) 2022-03-08T04:12:07.907 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-08T04:12:07.920 INFO:teuthology.task.internal:Opening connections... 2022-03-08T04:12:07.921 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi018.front.sepia.ceph.com 2022-03-08T04:12:07.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:12:07.976 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2022-03-08T04:12:07.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:12:08.025 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2022-03-08T04:12:08.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:12:08.078 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-08T04:12:08.087 DEBUG:teuthology.orchestra.run.smithi018:> uname -m 2022-03-08T04:12:08.094 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2022-03-08T04:12:08.095 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release 2022-03-08T04:12:08.144 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="Ubuntu" 2022-03-08T04:12:08.144 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T04:12:08.145 INFO:teuthology.orchestra.run.smithi018.stdout:ID=ubuntu 2022-03-08T04:12:08.145 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE=debian 2022-03-08T04:12:08.145 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T04:12:08.145 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="20.04" 2022-03-08T04:12:08.146 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T04:12:08.146 INFO:teuthology.orchestra.run.smithi018.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T04:12:08.146 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T04:12:08.146 INFO:teuthology.orchestra.run.smithi018.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T04:12:08.147 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_CODENAME=focal 2022-03-08T04:12:08.147 INFO:teuthology.orchestra.run.smithi018.stdout:UBUNTU_CODENAME=focal 2022-03-08T04:12:08.148 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2022-03-08T04:12:08.168 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2022-03-08T04:12:08.176 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2022-03-08T04:12:08.176 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2022-03-08T04:12:08.224 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Ubuntu" 2022-03-08T04:12:08.224 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T04:12:08.225 INFO:teuthology.orchestra.run.smithi084.stdout:ID=ubuntu 2022-03-08T04:12:08.225 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE=debian 2022-03-08T04:12:08.225 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T04:12:08.225 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="20.04" 2022-03-08T04:12:08.226 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T04:12:08.226 INFO:teuthology.orchestra.run.smithi084.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T04:12:08.226 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T04:12:08.226 INFO:teuthology.orchestra.run.smithi084.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T04:12:08.227 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_CODENAME=focal 2022-03-08T04:12:08.227 INFO:teuthology.orchestra.run.smithi084.stdout:UBUNTU_CODENAME=focal 2022-03-08T04:12:08.227 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2022-03-08T04:12:08.244 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2022-03-08T04:12:08.251 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2022-03-08T04:12:08.251 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2022-03-08T04:12:08.299 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="Ubuntu" 2022-03-08T04:12:08.299 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-08T04:12:08.299 INFO:teuthology.orchestra.run.smithi105.stdout:ID=ubuntu 2022-03-08T04:12:08.299 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE=debian 2022-03-08T04:12:08.300 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-08T04:12:08.300 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="20.04" 2022-03-08T04:12:08.300 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-08T04:12:08.300 INFO:teuthology.orchestra.run.smithi105.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-08T04:12:08.301 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-08T04:12:08.301 INFO:teuthology.orchestra.run.smithi105.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-08T04:12:08.301 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_CODENAME=focal 2022-03-08T04:12:08.301 INFO:teuthology.orchestra.run.smithi105.stdout:UBUNTU_CODENAME=focal 2022-03-08T04:12:08.302 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2022-03-08T04:12:08.319 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-08T04:12:08.333 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-08T04:12:08.352 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-08T04:12:08.352 DEBUG:teuthology.orchestra.run.smithi018:> test '!' -e /home/ubuntu/cephtest 2022-03-08T04:12:08.355 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2022-03-08T04:12:08.357 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2022-03-08T04:12:08.363 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-08T04:12:08.377 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-08T04:12:08.378 DEBUG:teuthology.orchestra.run.smithi018:> test -z $(ls -A /var/lib/ceph) 2022-03-08T04:12:08.404 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2022-03-08T04:12:08.407 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2022-03-08T04:12:08.421 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-08T04:12:08.500 INFO:teuthology.run_tasks:Running task kernel... 2022-03-08T04:12:08.523 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-08T04:12:08.524 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-08T04:12:08.525 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-08T04:12:08.525 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-08T04:12:08.526 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2022-03-08T04:12:08.526 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2022-03-08T04:12:08.527 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2022-03-08T04:12:08.534 INFO:teuthology.orchestra.run.smithi084.stdout:5.4.0-97-generic 2022-03-08T04:12:08.534 INFO:teuthology.task.kernel:Running kernel on smithi084: 5.4.0-97-generic 2022-03-08T04:12:08.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get clean 2022-03-08T04:12:08.536 INFO:teuthology.orchestra.run.smithi105.stdout:5.4.0-97-generic 2022-03-08T04:12:08.537 INFO:teuthology.orchestra.run.smithi018.stdout:5.4.0-97-generic 2022-03-08T04:12:08.537 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.4.0-97-generic 2022-03-08T04:12:08.537 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get clean 2022-03-08T04:12:08.538 INFO:teuthology.task.kernel:Running kernel on smithi018: 5.4.0-97-generic 2022-03-08T04:12:08.538 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get clean 2022-03-08T04:12:08.660 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get update 2022-03-08T04:12:08.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get update 2022-03-08T04:12:08.681 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2022-03-08T04:12:08.822 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T04:12:08.872 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T04:12:08.933 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:12:08.980 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:12:09.000 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:12:09.000 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-08T04:12:09.015 INFO:teuthology.orchestra.run.smithi105.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T04:12:09.055 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T04:12:09.063 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-08T04:12:09.135 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-08T04:12:09.172 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-08T04:12:09.256 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-08T04:12:09.270 INFO:teuthology.orchestra.run.smithi105.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-08T04:12:09.273 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-08T04:12:09.278 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T04:12:09.278 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-08T04:12:09.284 INFO:teuthology.orchestra.run.smithi105.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T04:12:09.284 INFO:teuthology.orchestra.run.smithi105.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-08T04:12:09.286 INFO:teuthology.orchestra.run.smithi105.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T04:12:09.286 INFO:teuthology.orchestra.run.smithi105.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T04:12:09.291 INFO:teuthology.orchestra.run.smithi105.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T04:12:09.299 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-08T04:12:09.301 INFO:teuthology.orchestra.run.smithi105.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T04:12:09.302 INFO:teuthology.orchestra.run.smithi105.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T04:12:09.303 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T04:12:09.304 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-08T04:12:09.311 INFO:teuthology.orchestra.run.smithi084.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T04:12:09.312 INFO:teuthology.orchestra.run.smithi084.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-08T04:12:09.313 INFO:teuthology.orchestra.run.smithi084.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T04:12:09.313 INFO:teuthology.orchestra.run.smithi084.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T04:12:09.319 INFO:teuthology.orchestra.run.smithi105.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T04:12:09.320 INFO:teuthology.orchestra.run.smithi105.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T04:12:09.321 INFO:teuthology.orchestra.run.smithi105.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T04:12:09.321 INFO:teuthology.orchestra.run.smithi105.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T04:12:09.322 INFO:teuthology.orchestra.run.smithi084.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T04:12:09.326 INFO:teuthology.orchestra.run.smithi084.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T04:12:09.327 INFO:teuthology.orchestra.run.smithi084.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T04:12:09.340 INFO:teuthology.orchestra.run.smithi084.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T04:12:09.340 INFO:teuthology.orchestra.run.smithi084.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T04:12:09.341 INFO:teuthology.orchestra.run.smithi084.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T04:12:09.341 INFO:teuthology.orchestra.run.smithi084.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T04:12:09.358 INFO:teuthology.orchestra.run.smithi105.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T04:12:09.393 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T04:12:09.401 INFO:teuthology.orchestra.run.smithi084.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-08T04:12:09.472 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-08T04:12:09.525 INFO:teuthology.orchestra.run.smithi084.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-08T04:12:09.531 INFO:teuthology.orchestra.run.smithi105.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-08T04:12:09.622 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-08T04:12:09.709 INFO:teuthology.orchestra.run.smithi105.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-08T04:12:09.779 INFO:teuthology.orchestra.run.smithi084.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-08T04:12:09.854 INFO:teuthology.orchestra.run.smithi105.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-08T04:12:09.870 INFO:teuthology.orchestra.run.smithi105.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T04:12:09.870 INFO:teuthology.orchestra.run.smithi105.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-08T04:12:09.900 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-08T04:12:09.903 INFO:teuthology.orchestra.run.smithi105.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T04:12:09.904 INFO:teuthology.orchestra.run.smithi105.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-08T04:12:09.907 INFO:teuthology.orchestra.run.smithi105.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T04:12:09.907 INFO:teuthology.orchestra.run.smithi105.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-08T04:12:09.911 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-08T04:12:09.916 INFO:teuthology.orchestra.run.smithi018.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-08T04:12:09.920 INFO:teuthology.orchestra.run.smithi084.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-08T04:12:09.926 INFO:teuthology.orchestra.run.smithi105.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-08T04:12:09.927 INFO:teuthology.orchestra.run.smithi018.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-08T04:12:09.927 INFO:teuthology.orchestra.run.smithi018.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-08T04:12:09.928 INFO:teuthology.orchestra.run.smithi018.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-08T04:12:09.937 INFO:teuthology.orchestra.run.smithi084.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T04:12:09.937 INFO:teuthology.orchestra.run.smithi084.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T04:12:09.938 INFO:teuthology.orchestra.run.smithi084.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-08T04:12:09.950 INFO:teuthology.orchestra.run.smithi105.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-08T04:12:09.955 INFO:teuthology.orchestra.run.smithi105.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T04:12:09.967 INFO:teuthology.orchestra.run.smithi084.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-08T04:12:09.972 INFO:teuthology.orchestra.run.smithi084.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T04:12:09.972 INFO:teuthology.orchestra.run.smithi084.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-08T04:12:09.984 INFO:teuthology.orchestra.run.smithi018.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-08T04:12:09.988 INFO:teuthology.orchestra.run.smithi018.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-08T04:12:09.988 INFO:teuthology.orchestra.run.smithi018.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-08T04:12:09.990 INFO:teuthology.orchestra.run.smithi105.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T04:12:09.990 INFO:teuthology.orchestra.run.smithi105.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T04:12:09.991 INFO:teuthology.orchestra.run.smithi105.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T04:12:09.992 INFO:teuthology.orchestra.run.smithi105.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T04:12:09.992 INFO:teuthology.orchestra.run.smithi105.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T04:12:09.996 INFO:teuthology.orchestra.run.smithi084.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-08T04:12:10.019 INFO:teuthology.orchestra.run.smithi105.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T04:12:10.020 INFO:teuthology.orchestra.run.smithi105.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T04:12:10.021 INFO:teuthology.orchestra.run.smithi018.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-08T04:12:10.021 INFO:teuthology.orchestra.run.smithi084.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-08T04:12:10.026 INFO:teuthology.orchestra.run.smithi084.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T04:12:10.044 INFO:teuthology.orchestra.run.smithi018.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-08T04:12:10.046 INFO:teuthology.orchestra.run.smithi018.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-08T04:12:10.051 INFO:teuthology.orchestra.run.smithi018.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-08T04:12:10.055 INFO:teuthology.orchestra.run.smithi084.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T04:12:10.055 INFO:teuthology.orchestra.run.smithi084.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T04:12:10.056 INFO:teuthology.orchestra.run.smithi084.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T04:12:10.056 INFO:teuthology.orchestra.run.smithi084.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T04:12:10.057 INFO:teuthology.orchestra.run.smithi084.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T04:12:10.084 INFO:teuthology.orchestra.run.smithi018.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-08T04:12:10.084 INFO:teuthology.orchestra.run.smithi018.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-08T04:12:10.085 INFO:teuthology.orchestra.run.smithi084.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T04:12:10.085 INFO:teuthology.orchestra.run.smithi018.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-08T04:12:10.086 INFO:teuthology.orchestra.run.smithi018.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-08T04:12:10.086 INFO:teuthology.orchestra.run.smithi084.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T04:12:10.100 INFO:teuthology.orchestra.run.smithi105.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T04:12:10.101 INFO:teuthology.orchestra.run.smithi105.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T04:12:10.102 INFO:teuthology.orchestra.run.smithi105.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T04:12:10.103 INFO:teuthology.orchestra.run.smithi105.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T04:12:10.154 INFO:teuthology.orchestra.run.smithi018.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-08T04:12:10.155 INFO:teuthology.orchestra.run.smithi018.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-08T04:12:10.167 INFO:teuthology.orchestra.run.smithi084.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T04:12:10.167 INFO:teuthology.orchestra.run.smithi084.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T04:12:10.168 INFO:teuthology.orchestra.run.smithi084.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T04:12:10.168 INFO:teuthology.orchestra.run.smithi084.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T04:12:10.181 INFO:teuthology.orchestra.run.smithi018.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-08T04:12:10.181 INFO:teuthology.orchestra.run.smithi018.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-08T04:12:10.185 INFO:teuthology.orchestra.run.smithi018.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-08T04:12:10.185 INFO:teuthology.orchestra.run.smithi018.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-08T04:12:10.206 INFO:teuthology.orchestra.run.smithi018.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-08T04:12:10.243 INFO:teuthology.orchestra.run.smithi018.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-08T04:12:10.250 INFO:teuthology.orchestra.run.smithi018.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-08T04:12:10.288 INFO:teuthology.orchestra.run.smithi018.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-08T04:12:10.289 INFO:teuthology.orchestra.run.smithi018.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-08T04:12:10.289 INFO:teuthology.orchestra.run.smithi018.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-08T04:12:10.290 INFO:teuthology.orchestra.run.smithi018.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-08T04:12:10.290 INFO:teuthology.orchestra.run.smithi018.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-08T04:12:10.321 INFO:teuthology.orchestra.run.smithi018.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-08T04:12:10.321 INFO:teuthology.orchestra.run.smithi018.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-08T04:12:10.402 INFO:teuthology.orchestra.run.smithi018.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-08T04:12:10.403 INFO:teuthology.orchestra.run.smithi018.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-08T04:12:10.404 INFO:teuthology.orchestra.run.smithi018.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-08T04:12:10.404 INFO:teuthology.orchestra.run.smithi018.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-08T04:12:12.293 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 10.0 MB in 2s (6,061 kB/s) 2022-03-08T04:12:12.447 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 10.0 MB in 2s (5,975 kB/s) 2022-03-08T04:12:13.320 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 10.0 MB in 2s (4,602 kB/s) 2022-03-08T04:12:13.469 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2022-03-08T04:12:13.493 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T04:12:13.542 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2022-03-08T04:12:13.564 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2022-03-08T04:12:13.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T04:12:13.629 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2022-03-08T04:12:13.739 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2022-03-08T04:12:13.739 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2022-03-08T04:12:13.802 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2022-03-08T04:12:13.803 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2022-03-08T04:12:13.887 INFO:teuthology.orchestra.run.smithi105.stdout:The following additional packages will be installed: 2022-03-08T04:12:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T04:12:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-08T04:12:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-08T04:12:13.889 INFO:teuthology.orchestra.run.smithi105.stdout:Suggested packages: 2022-03-08T04:12:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T04:12:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-08T04:12:13.932 INFO:teuthology.orchestra.run.smithi105.stdout:The following NEW packages will be installed: 2022-03-08T04:12:13.933 INFO:teuthology.orchestra.run.smithi105.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T04:12:13.933 INFO:teuthology.orchestra.run.smithi105.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-08T04:12:13.934 INFO:teuthology.orchestra.run.smithi105.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-08T04:12:13.934 INFO:teuthology.orchestra.run.smithi105.stdout: thermald upower usbmuxd 2022-03-08T04:12:13.940 INFO:teuthology.orchestra.run.smithi084.stdout:The following additional packages will be installed: 2022-03-08T04:12:13.940 INFO:teuthology.orchestra.run.smithi084.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T04:12:13.941 INFO:teuthology.orchestra.run.smithi084.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-08T04:12:13.941 INFO:teuthology.orchestra.run.smithi084.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-08T04:12:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Suggested packages: 2022-03-08T04:12:13.942 INFO:teuthology.orchestra.run.smithi084.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T04:12:13.942 INFO:teuthology.orchestra.run.smithi084.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-08T04:12:13.990 INFO:teuthology.orchestra.run.smithi084.stdout:The following NEW packages will be installed: 2022-03-08T04:12:13.991 INFO:teuthology.orchestra.run.smithi084.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T04:12:13.992 INFO:teuthology.orchestra.run.smithi084.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-08T04:12:13.992 INFO:teuthology.orchestra.run.smithi084.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-08T04:12:13.992 INFO:teuthology.orchestra.run.smithi084.stdout: thermald upower usbmuxd 2022-03-08T04:12:14.116 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-08T04:12:14.116 INFO:teuthology.orchestra.run.smithi105.stdout:Need to get 80.1 MB of archives. 2022-03-08T04:12:14.117 INFO:teuthology.orchestra.run.smithi105.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T04:12:14.117 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T04:12:14.170 INFO:teuthology.orchestra.run.smithi084.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-08T04:12:14.171 INFO:teuthology.orchestra.run.smithi084.stdout:Need to get 80.1 MB of archives. 2022-03-08T04:12:14.171 INFO:teuthology.orchestra.run.smithi084.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T04:12:14.171 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T04:12:14.383 INFO:teuthology.orchestra.run.smithi105.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T04:12:14.405 INFO:teuthology.orchestra.run.smithi105.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T04:12:14.406 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T04:12:14.441 INFO:teuthology.orchestra.run.smithi084.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T04:12:14.473 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T04:12:14.492 INFO:teuthology.orchestra.run.smithi105.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T04:12:14.496 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T04:12:14.504 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-08T04:12:14.557 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T04:12:14.559 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-08T04:12:14.584 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T04:12:14.585 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-08T04:12:14.652 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-08T04:12:14.828 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-08T04:12:14.829 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-08T04:12:14.981 INFO:teuthology.orchestra.run.smithi018.stdout:The following additional packages will be installed: 2022-03-08T04:12:14.982 INFO:teuthology.orchestra.run.smithi018.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T04:12:14.982 INFO:teuthology.orchestra.run.smithi018.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-08T04:12:14.982 INFO:teuthology.orchestra.run.smithi018.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-08T04:12:14.983 INFO:teuthology.orchestra.run.smithi018.stdout:Suggested packages: 2022-03-08T04:12:14.983 INFO:teuthology.orchestra.run.smithi018.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-08T04:12:14.983 INFO:teuthology.orchestra.run.smithi018.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-08T04:12:15.034 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2022-03-08T04:12:15.035 INFO:teuthology.orchestra.run.smithi018.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-08T04:12:15.035 INFO:teuthology.orchestra.run.smithi018.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-08T04:12:15.036 INFO:teuthology.orchestra.run.smithi018.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-08T04:12:15.036 INFO:teuthology.orchestra.run.smithi018.stdout: thermald upower usbmuxd 2022-03-08T04:12:15.216 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-08T04:12:15.216 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 80.1 MB of archives. 2022-03-08T04:12:15.216 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-08T04:12:15.217 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-08T04:12:15.347 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-08T04:12:15.451 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-08T04:12:15.486 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-08T04:12:15.500 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-08T04:12:15.509 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-08T04:12:15.600 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-08T04:12:15.601 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-08T04:12:15.609 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-08T04:12:15.720 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-08T04:12:16.497 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-08T04:12:16.773 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-08T04:12:17.385 INFO:teuthology.orchestra.run.smithi105.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-08T04:12:17.386 INFO:teuthology.orchestra.run.smithi105.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T04:12:17.386 INFO:teuthology.orchestra.run.smithi105.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T04:12:17.392 INFO:teuthology.orchestra.run.smithi105.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T04:12:17.394 INFO:teuthology.orchestra.run.smithi105.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T04:12:17.506 INFO:teuthology.orchestra.run.smithi084.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-08T04:12:17.507 INFO:teuthology.orchestra.run.smithi084.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T04:12:17.507 INFO:teuthology.orchestra.run.smithi084.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T04:12:17.513 INFO:teuthology.orchestra.run.smithi084.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T04:12:17.515 INFO:teuthology.orchestra.run.smithi084.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T04:12:17.805 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 80.1 MB in 3s (23.2 MB/s) 2022-03-08T04:12:17.913 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 80.1 MB in 4s (22.8 MB/s) 2022-03-08T04:12:18.023 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T04:12:18.110 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T04:12:18.547 INFO:teuthology.orchestra.run.smithi018.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-08T04:12:18.548 INFO:teuthology.orchestra.run.smithi018.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-08T04:12:18.617 INFO:teuthology.orchestra.run.smithi018.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-08T04:12:18.622 INFO:teuthology.orchestra.run.smithi018.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-08T04:12:18.624 INFO:teuthology.orchestra.run.smithi018.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-08T04:12:19.051 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 80.1 MB in 4s (22.3 MB/s) 2022-03-08T04:12:19.240 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-08T04:12:19.474 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T04:12:19.478 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T04:12:19.535 INFO:teuthology.orchestra.run.smithi084.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T04:12:19.540 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T04:12:19.603 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T04:12:19.665 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T04:12:19.956 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T04:12:19.969 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T04:12:20.001 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T04:12:20.015 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T04:12:20.023 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T04:12:20.059 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T04:12:20.351 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T04:12:20.354 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T04:12:20.365 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T04:12:20.367 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T04:12:20.409 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T04:12:20.412 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T04:12:20.729 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T04:12:20.740 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T04:12:20.742 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T04:12:20.754 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-08T04:12:20.755 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T04:12:20.759 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-08T04:12:20.796 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T04:12:20.815 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T04:12:20.945 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T04:12:21.123 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T04:12:21.134 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T04:12:21.137 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T04:12:21.149 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T04:12:21.181 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T04:12:21.192 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T04:12:21.472 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-08T04:12:21.478 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-08T04:12:21.484 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-08T04:12:21.486 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-08T04:12:21.492 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T04:12:21.499 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T04:12:21.528 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:21.531 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T04:12:21.542 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:21.850 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-08T04:12:21.865 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-08T04:12:21.900 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T04:12:22.262 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-08T04:12:22.275 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-08T04:12:22.312 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T04:12:22.698 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-08T04:12:22.713 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-08T04:12:22.756 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T04:12:23.092 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-08T04:12:23.106 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T04:12:23.150 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:23.718 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-08T04:12:23.728 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-08T04:12:23.734 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T04:12:23.744 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T04:12:23.834 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:23.844 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:24.289 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-08T04:12:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T04:12:24.338 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-08T04:12:24.339 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:24.350 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T04:12:24.387 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:25.425 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-08T04:12:25.439 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T04:12:25.535 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:26.037 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-08T04:12:26.052 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-08T04:12:26.120 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:30.324 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T04:12:30.341 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-08T04:12:30.382 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-08T04:12:30.494 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T04:12:30.511 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-08T04:12:30.552 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-08T04:12:30.713 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T04:12:30.729 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T04:12:30.771 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T04:12:30.846 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T04:12:30.860 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T04:12:30.907 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T04:12:31.083 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package thermald. 2022-03-08T04:12:31.099 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T04:12:31.141 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T04:12:31.377 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package thermald. 2022-03-08T04:12:31.391 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T04:12:31.435 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T04:12:31.707 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package upower. 2022-03-08T04:12:31.722 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T04:12:31.775 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T04:12:31.950 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package upower. 2022-03-08T04:12:31.966 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T04:12:32.027 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T04:12:32.029 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-08T04:12:32.044 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-08T04:12:32.112 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-08T04:12:32.190 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T04:12:32.205 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T04:12:32.265 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T04:12:32.450 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T04:12:32.463 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-08T04:12:32.464 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T04:12:32.478 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-08T04:12:32.509 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T04:12:32.521 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T04:12:32.538 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T04:12:32.662 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T04:12:32.779 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T04:12:32.800 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T04:12:32.905 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T04:12:32.907 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T04:12:32.925 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package thermald. 2022-03-08T04:12:32.939 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-08T04:12:32.983 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T04:12:33.033 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T04:12:33.049 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T04:12:33.159 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T04:12:33.174 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T04:12:33.285 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T04:12:33.300 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T04:12:33.411 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T04:12:33.553 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T04:12:33.591 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package upower. 2022-03-08T04:12:33.608 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-08T04:12:33.659 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-08T04:12:33.735 INFO:teuthology.orchestra.run.smithi105.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T04:12:33.801 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T04:12:33.991 INFO:teuthology.orchestra.run.smithi084.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T04:12:33.997 INFO:teuthology.orchestra.run.smithi105.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T04:12:34.005 INFO:teuthology.orchestra.run.smithi105.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T04:12:34.005 INFO:teuthology.orchestra.run.smithi105.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T04:12:34.038 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T04:12:34.133 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package usbmuxd. 2022-03-08T04:12:34.147 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-08T04:12:34.191 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T04:12:34.205 INFO:teuthology.orchestra.run.smithi084.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T04:12:34.218 INFO:teuthology.orchestra.run.smithi084.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T04:12:34.218 INFO:teuthology.orchestra.run.smithi084.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T04:12:34.435 INFO:teuthology.orchestra.run.smithi105.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T04:12:34.496 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-08T04:12:34.646 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-08T04:12:34.661 INFO:teuthology.orchestra.run.smithi105.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T04:12:34.688 INFO:teuthology.orchestra.run.smithi084.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T04:12:34.723 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T04:12:34.805 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-08T04:12:34.923 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-08T04:12:34.931 INFO:teuthology.orchestra.run.smithi084.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T04:12:34.976 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T04:12:35.039 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T04:12:35.040 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T04:12:35.066 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-08T04:12:35.167 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-08T04:12:35.282 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T04:12:35.283 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T04:12:35.318 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-08T04:12:35.520 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:35.761 INFO:teuthology.orchestra.run.smithi018.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-08T04:12:35.772 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:35.919 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-08T04:12:36.400 INFO:teuthology.orchestra.run.smithi018.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-08T04:12:36.410 INFO:teuthology.orchestra.run.smithi018.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-08T04:12:36.410 INFO:teuthology.orchestra.run.smithi018.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-08T04:12:36.778 INFO:teuthology.orchestra.run.smithi105.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-08T04:12:36.778 INFO:teuthology.orchestra.run.smithi105.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-08T04:12:36.937 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:36.961 INFO:teuthology.orchestra.run.smithi018.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-08T04:12:37.031 INFO:teuthology.orchestra.run.smithi084.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-08T04:12:37.031 INFO:teuthology.orchestra.run.smithi084.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-08T04:12:37.180 INFO:teuthology.orchestra.run.smithi018.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-08T04:12:37.190 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:37.216 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-08T04:12:37.533 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T04:12:37.533 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-08T04:12:38.004 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:38.517 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:39.323 INFO:teuthology.orchestra.run.smithi018.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-08T04:12:39.324 INFO:teuthology.orchestra.run.smithi018.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-08T04:12:39.490 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:39.582 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-08T04:12:39.708 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T04:12:39.835 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-08T04:12:39.977 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T04:12:40.874 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:42.443 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-08T04:12:42.594 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-08T04:12:43.397 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T04:12:44.074 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T04:12:44.107 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T04:12:44.182 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:44.732 INFO:teuthology.orchestra.run.smithi105.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T04:12:44.732 INFO:teuthology.orchestra.run.smithi105.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-08T04:12:44.784 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T04:12:44.909 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:45.110 INFO:teuthology.orchestra.run.smithi084.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T04:12:45.110 INFO:teuthology.orchestra.run.smithi084.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-08T04:12:46.624 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T04:12:47.734 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-08T04:12:47.843 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-08T04:12:48.052 INFO:teuthology.orchestra.run.smithi018.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-08T04:12:48.053 INFO:teuthology.orchestra.run.smithi018.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-08T04:13:08.433 INFO:teuthology.orchestra.run.smithi084.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T04:13:08.434 INFO:teuthology.orchestra.run.smithi084.stdout:Sourcing file `/etc/default/grub' 2022-03-08T04:13:08.442 INFO:teuthology.orchestra.run.smithi084.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T04:13:08.478 INFO:teuthology.orchestra.run.smithi084.stdout:Generating grub configuration file ... 2022-03-08T04:13:08.768 INFO:teuthology.orchestra.run.smithi084.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-08T04:13:08.769 INFO:teuthology.orchestra.run.smithi105.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T04:13:08.769 INFO:teuthology.orchestra.run.smithi105.stdout:Sourcing file `/etc/default/grub' 2022-03-08T04:13:08.778 INFO:teuthology.orchestra.run.smithi105.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T04:13:08.796 INFO:teuthology.orchestra.run.smithi105.stdout:Generating grub configuration file ... 2022-03-08T04:13:08.800 INFO:teuthology.orchestra.run.smithi084.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-08T04:13:09.110 INFO:teuthology.orchestra.run.smithi105.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-08T04:13:09.136 INFO:teuthology.orchestra.run.smithi105.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-08T04:13:09.221 INFO:teuthology.orchestra.run.smithi084.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T04:13:09.233 INFO:teuthology.orchestra.run.smithi084.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T04:13:09.510 INFO:teuthology.orchestra.run.smithi105.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T04:13:09.520 INFO:teuthology.orchestra.run.smithi105.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T04:13:10.896 INFO:teuthology.orchestra.run.smithi084.stdout:done 2022-03-08T04:13:11.245 DEBUG:teuthology.orchestra.run.smithi084:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T04:13:11.402 INFO:teuthology.orchestra.run.smithi084.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T04:13:11.402 INFO:teuthology.orchestra.run.smithi084.stdout:Status: install ok installed 2022-03-08T04:13:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Priority: optional 2022-03-08T04:13:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Section: kernel 2022-03-08T04:13:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Installed-Size: 19 2022-03-08T04:13:11.403 INFO:teuthology.orchestra.run.smithi084.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T04:13:11.404 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: amd64 2022-03-08T04:13:11.404 INFO:teuthology.orchestra.run.smithi084.stdout:Source: linux-meta-hwe-5.13 2022-03-08T04:13:11.404 INFO:teuthology.orchestra.run.smithi084.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-08T04:13:11.404 INFO:teuthology.orchestra.run.smithi084.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T04:13:11.404 INFO:teuthology.orchestra.run.smithi084.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T04:13:11.405 INFO:teuthology.orchestra.run.smithi084.stdout:Recommends: thermald 2022-03-08T04:13:11.405 INFO:teuthology.orchestra.run.smithi084.stdout:Description: Generic Linux kernel image 2022-03-08T04:13:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: This package will always depend on the latest generic kernel image 2022-03-08T04:13:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: available. 2022-03-08T04:13:11.406 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-08T04:13:11.407 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T04:13:11.407 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T04:13:11.407 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-08T04:13:11.407 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T04:13:11.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get clean 2022-03-08T04:13:11.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt-get update 2022-03-08T04:13:11.674 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T04:13:11.788 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:13:11.870 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T04:13:11.953 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T04:13:12.122 INFO:teuthology.orchestra.run.smithi105.stdout:done 2022-03-08T04:13:12.156 INFO:teuthology.orchestra.run.smithi018.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-08T04:13:12.156 INFO:teuthology.orchestra.run.smithi018.stdout:Sourcing file `/etc/default/grub' 2022-03-08T04:13:12.166 INFO:teuthology.orchestra.run.smithi018.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T04:13:12.184 INFO:teuthology.orchestra.run.smithi018.stdout:Generating grub configuration file ... 2022-03-08T04:13:12.463 DEBUG:teuthology.orchestra.run.smithi105:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T04:13:12.502 INFO:teuthology.orchestra.run.smithi105.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T04:13:12.502 INFO:teuthology.orchestra.run.smithi105.stdout:Status: install ok installed 2022-03-08T04:13:12.502 INFO:teuthology.orchestra.run.smithi105.stdout:Priority: optional 2022-03-08T04:13:12.503 INFO:teuthology.orchestra.run.smithi105.stdout:Section: kernel 2022-03-08T04:13:12.503 INFO:teuthology.orchestra.run.smithi105.stdout:Installed-Size: 19 2022-03-08T04:13:12.503 INFO:teuthology.orchestra.run.smithi105.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T04:13:12.503 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: amd64 2022-03-08T04:13:12.504 INFO:teuthology.orchestra.run.smithi105.stdout:Source: linux-meta-hwe-5.13 2022-03-08T04:13:12.504 INFO:teuthology.orchestra.run.smithi105.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-08T04:13:12.504 INFO:teuthology.orchestra.run.smithi105.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T04:13:12.504 INFO:teuthology.orchestra.run.smithi105.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T04:13:12.505 INFO:teuthology.orchestra.run.smithi105.stdout:Recommends: thermald 2022-03-08T04:13:12.505 INFO:teuthology.orchestra.run.smithi105.stdout:Description: Generic Linux kernel image 2022-03-08T04:13:12.505 INFO:teuthology.orchestra.run.smithi105.stdout: This package will always depend on the latest generic kernel image 2022-03-08T04:13:12.505 INFO:teuthology.orchestra.run.smithi105.stdout: available. 2022-03-08T04:13:12.506 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-08T04:13:12.506 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T04:13:12.506 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T04:13:12.506 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-08T04:13:12.507 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T04:13:12.507 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get clean 2022-03-08T04:13:12.540 INFO:teuthology.orchestra.run.smithi018.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-08T04:13:12.572 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get update 2022-03-08T04:13:12.575 INFO:teuthology.orchestra.run.smithi018.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-08T04:13:12.838 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:13:12.838 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T04:13:12.909 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T04:13:12.991 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T04:13:13.184 INFO:teuthology.orchestra.run.smithi018.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T04:13:13.196 INFO:teuthology.orchestra.run.smithi018.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T04:13:13.229 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2022-03-08T04:13:13.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T04:13:13.322 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2022-03-08T04:13:13.477 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2022-03-08T04:13:13.477 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2022-03-08T04:13:13.645 INFO:teuthology.orchestra.run.smithi084.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-08T04:13:13.646 INFO:teuthology.orchestra.run.smithi084.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-08T04:13:13.647 DEBUG:teuthology.orchestra.run.smithi084:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T04:13:13.667 INFO:teuthology.orchestra.run.smithi084.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T04:13:13.668 INFO:teuthology.orchestra.run.smithi084.stdout:Status: install ok installed 2022-03-08T04:13:13.668 INFO:teuthology.orchestra.run.smithi084.stdout:Priority: optional 2022-03-08T04:13:13.668 INFO:teuthology.orchestra.run.smithi084.stdout:Section: kernel 2022-03-08T04:13:13.668 INFO:teuthology.orchestra.run.smithi084.stdout:Installed-Size: 19 2022-03-08T04:13:13.669 INFO:teuthology.orchestra.run.smithi084.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T04:13:13.669 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: amd64 2022-03-08T04:13:13.669 INFO:teuthology.orchestra.run.smithi084.stdout:Source: linux-meta-hwe-5.13 2022-03-08T04:13:13.669 INFO:teuthology.orchestra.run.smithi084.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-08T04:13:13.670 INFO:teuthology.orchestra.run.smithi084.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T04:13:13.670 INFO:teuthology.orchestra.run.smithi084.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T04:13:13.670 INFO:teuthology.orchestra.run.smithi084.stdout:Recommends: thermald 2022-03-08T04:13:13.670 INFO:teuthology.orchestra.run.smithi084.stdout:Description: Generic Linux kernel image 2022-03-08T04:13:13.671 INFO:teuthology.orchestra.run.smithi084.stdout: This package will always depend on the latest generic kernel image 2022-03-08T04:13:13.671 INFO:teuthology.orchestra.run.smithi084.stdout: available. 2022-03-08T04:13:13.671 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2022-03-08T04:13:13.678 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.X25CqdSAtU 2022-03-08T04:13:13.678 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub/grub.cfg /tmp/tmp.X25CqdSAtU 2022-03-08T04:13:13.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.X25CqdSAtU 2022-03-08T04:13:13.777 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.X25CqdSAtU is 8KB 2022-03-08T04:13:13.792 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.X25CqdSAtU 2022-03-08T04:13:13.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T04:13:13.856 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:13:13.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T04:13:13.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T04:13:13.892 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-08T04:13:13.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo update-grub 2022-03-08T04:13:14.383 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2022-03-08T04:13:14.407 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T04:13:14.472 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2022-03-08T04:13:14.625 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2022-03-08T04:13:14.626 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2022-03-08T04:13:14.796 INFO:teuthology.orchestra.run.smithi105.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-08T04:13:14.796 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-08T04:13:14.798 DEBUG:teuthology.orchestra.run.smithi105:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T04:13:14.818 INFO:teuthology.orchestra.run.smithi105.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T04:13:14.819 INFO:teuthology.orchestra.run.smithi105.stdout:Status: install ok installed 2022-03-08T04:13:14.819 INFO:teuthology.orchestra.run.smithi105.stdout:Priority: optional 2022-03-08T04:13:14.819 INFO:teuthology.orchestra.run.smithi105.stdout:Section: kernel 2022-03-08T04:13:14.819 INFO:teuthology.orchestra.run.smithi105.stdout:Installed-Size: 19 2022-03-08T04:13:14.820 INFO:teuthology.orchestra.run.smithi105.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T04:13:14.820 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: amd64 2022-03-08T04:13:14.820 INFO:teuthology.orchestra.run.smithi105.stdout:Source: linux-meta-hwe-5.13 2022-03-08T04:13:14.820 INFO:teuthology.orchestra.run.smithi105.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-08T04:13:14.820 INFO:teuthology.orchestra.run.smithi105.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T04:13:14.821 INFO:teuthology.orchestra.run.smithi105.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T04:13:14.821 INFO:teuthology.orchestra.run.smithi105.stdout:Recommends: thermald 2022-03-08T04:13:14.821 INFO:teuthology.orchestra.run.smithi105.stdout:Description: Generic Linux kernel image 2022-03-08T04:13:14.821 INFO:teuthology.orchestra.run.smithi105.stdout: This package will always depend on the latest generic kernel image 2022-03-08T04:13:14.821 INFO:teuthology.orchestra.run.smithi105.stdout: available. 2022-03-08T04:13:14.822 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2022-03-08T04:13:14.828 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.cSJZMmTEyV 2022-03-08T04:13:14.828 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub/grub.cfg /tmp/tmp.cSJZMmTEyV 2022-03-08T04:13:14.844 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.cSJZMmTEyV 2022-03-08T04:13:14.928 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.cSJZMmTEyV is 8KB 2022-03-08T04:13:14.941 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.cSJZMmTEyV 2022-03-08T04:13:14.947 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T04:13:14.960 INFO:teuthology.orchestra.run.smithi018.stdout:done 2022-03-08T04:13:14.998 INFO:teuthology.orchestra.run.smithi084.stderr:Sourcing file `/etc/default/grub' 2022-03-08T04:13:15.000 INFO:teuthology.orchestra.run.smithi084.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T04:13:15.010 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:13:15.010 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T04:13:15.011 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T04:13:15.018 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2022-03-08T04:13:15.049 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-08T04:13:15.050 DEBUG:teuthology.orchestra.run.smithi105:> sudo update-grub 2022-03-08T04:13:15.300 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-08T04:13:15.312 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-08T04:13:15.317 DEBUG:teuthology.orchestra.run.smithi018:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T04:13:15.346 INFO:teuthology.orchestra.run.smithi018.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T04:13:15.346 INFO:teuthology.orchestra.run.smithi018.stdout:Status: install ok installed 2022-03-08T04:13:15.347 INFO:teuthology.orchestra.run.smithi018.stdout:Priority: optional 2022-03-08T04:13:15.347 INFO:teuthology.orchestra.run.smithi018.stdout:Section: kernel 2022-03-08T04:13:15.347 INFO:teuthology.orchestra.run.smithi018.stdout:Installed-Size: 19 2022-03-08T04:13:15.347 INFO:teuthology.orchestra.run.smithi018.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T04:13:15.348 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: amd64 2022-03-08T04:13:15.348 INFO:teuthology.orchestra.run.smithi018.stdout:Source: linux-meta-hwe-5.13 2022-03-08T04:13:15.348 INFO:teuthology.orchestra.run.smithi018.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-08T04:13:15.348 INFO:teuthology.orchestra.run.smithi018.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T04:13:15.349 INFO:teuthology.orchestra.run.smithi018.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T04:13:15.349 INFO:teuthology.orchestra.run.smithi018.stdout:Recommends: thermald 2022-03-08T04:13:15.349 INFO:teuthology.orchestra.run.smithi018.stdout:Description: Generic Linux kernel image 2022-03-08T04:13:15.349 INFO:teuthology.orchestra.run.smithi018.stdout: This package will always depend on the latest generic kernel image 2022-03-08T04:13:15.349 INFO:teuthology.orchestra.run.smithi018.stdout: available. 2022-03-08T04:13:15.350 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-08T04:13:15.351 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-08T04:13:15.351 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-08T04:13:15.351 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-08T04:13:15.351 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi018.front.sepia.ceph.com, path=None, version=distro) 2022-03-08T04:13:15.352 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get clean 2022-03-08T04:13:15.408 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2022-03-08T04:13:15.551 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T04:13:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T04:13:15.657 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:13:15.665 INFO:teuthology.orchestra.run.smithi084.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T04:13:15.740 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T04:13:15.822 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T04:13:16.242 INFO:teuthology.orchestra.run.smithi105.stderr:Sourcing file `/etc/default/grub' 2022-03-08T04:13:16.244 INFO:teuthology.orchestra.run.smithi105.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T04:13:16.262 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2022-03-08T04:13:16.357 INFO:teuthology.orchestra.run.smithi084.stderr:done 2022-03-08T04:13:16.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2022-03-08T04:13:16.497 INFO:teuthology.orchestra.run.smithi105.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-08T04:13:16.509 INFO:teuthology.orchestra.run.smithi105.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-08T04:13:16.855 INFO:teuthology.orchestra.run.smithi105.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T04:13:16.867 INFO:teuthology.orchestra.run.smithi105.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T04:13:17.125 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-08T04:13:17.147 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-08T04:13:17.215 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-08T04:13:17.394 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-08T04:13:17.395 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-08T04:13:17.586 INFO:teuthology.orchestra.run.smithi018.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-08T04:13:17.587 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-08T04:13:17.588 DEBUG:teuthology.orchestra.run.smithi018:> dpkg -s linux-image-generic-hwe-20.04 2022-03-08T04:13:17.611 INFO:teuthology.orchestra.run.smithi018.stdout:Package: linux-image-generic-hwe-20.04 2022-03-08T04:13:17.612 INFO:teuthology.orchestra.run.smithi018.stdout:Status: install ok installed 2022-03-08T04:13:17.612 INFO:teuthology.orchestra.run.smithi018.stdout:Priority: optional 2022-03-08T04:13:17.612 INFO:teuthology.orchestra.run.smithi018.stdout:Section: kernel 2022-03-08T04:13:17.613 INFO:teuthology.orchestra.run.smithi018.stdout:Installed-Size: 19 2022-03-08T04:13:17.613 INFO:teuthology.orchestra.run.smithi018.stdout:Maintainer: Ubuntu Kernel Team 2022-03-08T04:13:17.613 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: amd64 2022-03-08T04:13:17.613 INFO:teuthology.orchestra.run.smithi018.stdout:Source: linux-meta-hwe-5.13 2022-03-08T04:13:17.614 INFO:teuthology.orchestra.run.smithi018.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-08T04:13:17.614 INFO:teuthology.orchestra.run.smithi018.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-08T04:13:17.614 INFO:teuthology.orchestra.run.smithi018.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-08T04:13:17.614 INFO:teuthology.orchestra.run.smithi018.stdout:Recommends: thermald 2022-03-08T04:13:17.615 INFO:teuthology.orchestra.run.smithi018.stdout:Description: Generic Linux kernel image 2022-03-08T04:13:17.615 INFO:teuthology.orchestra.run.smithi018.stdout: This package will always depend on the latest generic kernel image 2022-03-08T04:13:17.615 INFO:teuthology.orchestra.run.smithi018.stdout: available. 2022-03-08T04:13:17.616 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2022-03-08T04:13:17.622 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.5gktyoOrCb 2022-03-08T04:13:17.623 DEBUG:teuthology.orchestra.run.smithi018:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5gktyoOrCb 2022-03-08T04:13:17.681 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.5gktyoOrCb 2022-03-08T04:13:17.719 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.5gktyoOrCb is 8KB 2022-03-08T04:13:17.733 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.5gktyoOrCb 2022-03-08T04:13:17.740 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-08T04:13:17.800 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:13:17.800 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-08T04:13:17.801 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-08T04:13:17.848 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-08T04:13:17.849 DEBUG:teuthology.orchestra.run.smithi018:> sudo update-grub 2022-03-08T04:13:18.460 INFO:teuthology.orchestra.run.smithi105.stderr:done 2022-03-08T04:13:18.462 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2022-03-08T04:13:19.137 INFO:teuthology.orchestra.run.smithi018.stderr:Sourcing file `/etc/default/grub' 2022-03-08T04:13:19.146 INFO:teuthology.orchestra.run.smithi018.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-08T04:13:19.158 INFO:teuthology.orchestra.run.smithi018.stderr:Generating grub configuration file ... 2022-03-08T04:13:19.415 INFO:teuthology.orchestra.run.smithi018.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-08T04:13:19.426 INFO:teuthology.orchestra.run.smithi018.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-08T04:13:19.787 INFO:teuthology.orchestra.run.smithi018.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-08T04:13:19.799 INFO:teuthology.orchestra.run.smithi018.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-08T04:13:20.488 INFO:teuthology.orchestra.run.smithi018.stderr:done 2022-03-08T04:13:20.489 DEBUG:teuthology.orchestra.run.smithi018:> sudo shutdown -r now 2022-03-08T04:13:46.366 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-08T04:13:46.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-03-08T04:13:46.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:13:48.466 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-08T04:13:48.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2022-03-08T04:13:48.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:13:50.494 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-08T04:13:50.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-03-08T04:13:50.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:14:04.751 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2022-03-08T04:14:06.795 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2022-03-08T04:14:08.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.18 2022-03-08T04:14:34.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-03-08T04:14:34.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:14:36.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2022-03-08T04:14:36.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:14:37.835 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2022-03-08T04:14:38.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-03-08T04:14:38.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:14:39.851 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2022-03-08T04:14:54.317 DEBUG:teuthology.orchestra.run.smithi018:> true 2022-03-08T04:14:54.889 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-03-08T04:14:54.890 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-30-generic"... 2022-03-08T04:14:54.890 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2022-03-08T04:14:54.895 INFO:teuthology.orchestra.run.smithi018.stdout:5.13.0-30-generic 2022-03-08T04:14:54.896 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-08T04:14:54.896 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T04:14:54.896 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T04:14:55.897 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-08T04:14:55.897 DEBUG:teuthology.orchestra.run.smithi018:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T04:14:55.964 INFO:teuthology.orchestra.run.smithi018.stdout:ttyS1 2022-03-08T04:14:55.987 DEBUG:teuthology.parallel:result is None 2022-03-08T04:15:07.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-03-08T04:15:07.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:15:07.992 DEBUG:teuthology.orchestra.run.smithi084:> true 2022-03-08T04:15:08.468 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-03-08T04:15:08.468 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-30-generic"... 2022-03-08T04:15:08.469 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2022-03-08T04:15:08.473 INFO:teuthology.orchestra.run.smithi084.stdout:5.13.0-30-generic 2022-03-08T04:15:08.474 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-08T04:15:08.474 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T04:15:08.474 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T04:15:09.475 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-08T04:15:09.476 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T04:15:09.540 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2022-03-08T04:15:09.562 DEBUG:teuthology.parallel:result is None 2022-03-08T04:15:09.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2022-03-08T04:15:09.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:15:09.946 DEBUG:teuthology.orchestra.run.smithi105:> true 2022-03-08T04:15:10.484 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2022-03-08T04:15:10.484 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-30-generic"... 2022-03-08T04:15:10.485 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2022-03-08T04:15:10.491 INFO:teuthology.orchestra.run.smithi105.stdout:5.13.0-30-generic 2022-03-08T04:15:10.492 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-08T04:15:10.492 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-08T04:15:10.492 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-08T04:15:11.493 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-08T04:15:11.494 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-08T04:15:11.566 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2022-03-08T04:15:11.593 DEBUG:teuthology.parallel:result is None 2022-03-08T04:15:11.594 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-08T04:15:11.602 INFO:teuthology.task.internal:Creating test directory... 2022-03-08T04:15:11.602 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T04:15:11.605 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T04:15:11.608 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-08T04:15:11.615 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-08T04:15:11.642 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-08T04:15:11.675 INFO:teuthology.task.internal:Creating archive directory... 2022-03-08T04:15:11.676 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T04:15:11.679 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T04:15:11.681 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-08T04:15:11.708 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-08T04:15:11.719 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-08T04:15:11.719 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T04:15:11.722 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T04:15:11.725 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-08T04:15:11.747 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T04:15:11.748 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T04:15:11.752 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T04:15:11.755 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T04:15:11.758 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T04:15:11.762 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-08T04:15:11.764 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-08T04:15:11.773 INFO:teuthology.task.internal:Configuring sudo... 2022-03-08T04:15:11.773 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T04:15:11.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T04:15:11.805 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-08T04:15:11.821 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-08T04:15:11.831 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-08T04:15:11.832 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T04:15:11.834 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T04:15:11.837 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-08T04:15:11.844 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T04:15:11.850 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T04:15:11.900 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:15:11.900 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T04:15:11.925 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T04:15:11.933 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T04:15:11.983 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:15:11.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T04:15:12.006 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-08T04:15:12.014 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-08T04:15:12.062 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:15:12.062 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-08T04:15:12.081 DEBUG:teuthology.orchestra.run.smithi018:> sudo service rsyslog restart 2022-03-08T04:15:12.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2022-03-08T04:15:12.087 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2022-03-08T04:15:12.134 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-08T04:15:12.142 INFO:teuthology.task.internal:Starting timer... 2022-03-08T04:15:12.142 INFO:teuthology.run_tasks:Running task pcp... 2022-03-08T04:15:12.162 INFO:teuthology.run_tasks:Running task selinux... 2022-03-08T04:15:12.194 DEBUG:teuthology.task.selinux:Excluding smithi018: OS 'ubuntu' does not support SELinux 2022-03-08T04:15:12.195 DEBUG:teuthology.task.selinux:Excluding smithi084: OS 'ubuntu' does not support SELinux 2022-03-08T04:15:12.195 DEBUG:teuthology.task.selinux:Excluding smithi105: OS 'ubuntu' does not support SELinux 2022-03-08T04:15:12.196 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-08T04:15:12.196 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-08T04:15:12.196 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-08T04:15:12.197 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-08T04:15:12.228 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-08T04:15:12.229 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-08T04:15:12.249 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-08T04:15:12.251 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi018.front.sepia.ceph.com,smithi084.front.sepia.ceph.com,smithi105.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-08T04:18:20.269 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi018.front.sepia.ceph.com'), Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com')] 2022-03-08T04:18:20.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-03-08T04:18:20.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:18:20.328 DEBUG:teuthology.orchestra.run.smithi018:> true 2022-03-08T04:18:20.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-03-08T04:18:20.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:18:20.508 DEBUG:teuthology.orchestra.run.smithi084:> true 2022-03-08T04:18:20.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2022-03-08T04:18:20.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-08T04:18:20.678 DEBUG:teuthology.orchestra.run.smithi105:> true 2022-03-08T04:18:20.797 INFO:teuthology.run_tasks:Running task clock... 2022-03-08T04:18:20.810 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-08T04:18:20.811 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T04:18:20.813 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T04:18:20.817 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T04:18:20.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T04:18:20.820 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-08T04:18:20.821 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-08T04:18:20.847 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T04:18:20.848 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: Command line: ntpd -gq 2022-03-08T04:18:20.849 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: proto: precision = 0.072 usec (-24) 2022-03-08T04:18:20.849 INFO:teuthology.orchestra.run.smithi018.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T04:18:20.850 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T04:18:20.850 INFO:teuthology.orchestra.run.smithi018.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T04:18:20.851 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T04:18:20.852 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T04:18:20.852 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T04:18:20.852 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T04:18:20.852 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: Listen normally on 3 ens1f0 172.21.15.18:123 2022-03-08T04:18:20.853 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: Listen normally on 4 lo [::1]:123 2022-03-08T04:18:20.853 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1362%4]:123 2022-03-08T04:18:20.853 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:20 ntpd[8940]: Listening on routing socket on fd #22 for interface updates 2022-03-08T04:18:20.854 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T04:18:20.855 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: Command line: ntpd -gq 2022-03-08T04:18:20.856 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: proto: precision = 0.074 usec (-24) 2022-03-08T04:18:20.856 INFO:teuthology.orchestra.run.smithi084.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T04:18:20.857 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T04:18:20.857 INFO:teuthology.orchestra.run.smithi084.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T04:18:20.858 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T04:18:20.859 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T04:18:20.859 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T04:18:20.859 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T04:18:20.859 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: Listen normally on 3 enp3s0f1 172.21.15.84:123 2022-03-08T04:18:20.860 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: Listen normally on 4 lo [::1]:123 2022-03-08T04:18:20.860 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d751%5]:123 2022-03-08T04:18:20.860 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:20 ntpd[9021]: Listening on routing socket on fd #22 for interface updates 2022-03-08T04:18:20.861 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-08T04:18:20.861 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: Command line: ntpd -gq 2022-03-08T04:18:20.862 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: proto: precision = 0.088 usec (-23) 2022-03-08T04:18:20.862 INFO:teuthology.orchestra.run.smithi105.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T04:18:20.863 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-08T04:18:20.863 INFO:teuthology.orchestra.run.smithi105.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-08T04:18:20.864 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: restrict ::: KOD does nothing without LIMITED. 2022-03-08T04:18:20.864 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: Listen and drop on 0 v6wildcard [::]:123 2022-03-08T04:18:20.865 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-08T04:18:20.865 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: Listen normally on 2 lo 127.0.0.1:123 2022-03-08T04:18:20.865 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: Listen normally on 3 enp3s0f1 172.21.15.105:123 2022-03-08T04:18:20.865 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: Listen normally on 4 lo [::1]:123 2022-03-08T04:18:20.866 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d429%5]:123 2022-03-08T04:18:20.866 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:20 ntpd[9007]: Listening on routing socket on fd #22 for interface updates 2022-03-08T04:18:21.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:21.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:21.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.853 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.853 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:21.853 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:21 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:21.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:21.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:21.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:21.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:21 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:21.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:21.864 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:21.864 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:21.864 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:21.864 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:21.865 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:21 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:22.849 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:22.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:22.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:22.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:22.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:22.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:22.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:22.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:22.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:22.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:22.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:22.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:22 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:22.853 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:22 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:22.853 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:22 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:22.854 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:22 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:22.854 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:22 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:22.854 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:22 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:22.854 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:22 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:22.857 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:22 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:22.857 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:22 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:22.858 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:22 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:22.858 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:22 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:22.858 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:22 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:22.858 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:22 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:23.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:23.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:23.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:23.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:23 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:23.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:23.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:23.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:23.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:23.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.859 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:23.859 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:23 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:23.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:23.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:23.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:23.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:23.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:23.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:23 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:24.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:24.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:24.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.853 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:24.853 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:24 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:24.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:24 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:24 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:24 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:24 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:24 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:24.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:24 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:24.859 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.859 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:24.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:24.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:24.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:24.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:24.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:24 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:25.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:25.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:25.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:25.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:25 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:25.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:25.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:25.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:25.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:25.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.859 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.859 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:25.859 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:25 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:25.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:25.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:25.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:25.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:25.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:25.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:25 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:26.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:26.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:26.852 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.853 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.853 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:26.853 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:26 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:26.854 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:26 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.854 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:26 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:26 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:26 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:26 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:26.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:26 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:26.857 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.858 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.858 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.858 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.858 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.859 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.859 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:26.859 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:26.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:26.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:26.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:26.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:26 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:27.849 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:27 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.849 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:27 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:27 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:27 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.850 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:27 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:27 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:27 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:27.851 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:27 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:27.852 INFO:teuthology.orchestra.run.smithi018.stdout: 8 Mar 04:18:27 ntpd[8940]: ntpd: time slew -0.000250 s 2022-03-08T04:18:27.852 INFO:teuthology.orchestra.run.smithi018.stdout:ntpd: time slew -0.000250s 2022-03-08T04:18:27.854 INFO:teuthology.orchestra.run.smithi018.stderr: 8 Mar 04:18:27 ntpd[8940]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T04:18:27.855 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.856 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.857 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:27.858 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:27.858 INFO:teuthology.orchestra.run.smithi084.stdout: 8 Mar 04:18:27 ntpd[9021]: ntpd: time slew -0.000679 s 2022-03-08T04:18:27.858 INFO:teuthology.orchestra.run.smithi084.stdout:ntpd: time slew -0.000679s 2022-03-08T04:18:27.859 INFO:teuthology.orchestra.run.smithi084.stderr: 8 Mar 04:18:27 ntpd[9021]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T04:18:27.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:27 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:27 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.860 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:27 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:27 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:27 ntpd[9007]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-08T04:18:27.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:27 ntpd[9007]: can't open /var/log/ntpstats/rawstats.20220308: Permission denied 2022-03-08T04:18:27.861 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:27 ntpd[9007]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-08T04:18:27.862 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:27 ntpd[9007]: can't open /var/log/ntpstats/peerstats.20220308: Permission denied 2022-03-08T04:18:27.862 INFO:teuthology.orchestra.run.smithi105.stdout: 8 Mar 04:18:27 ntpd[9007]: ntpd: time slew -0.002158 s 2022-03-08T04:18:27.862 INFO:teuthology.orchestra.run.smithi105.stdout:ntpd: time slew -0.002158s 2022-03-08T04:18:27.863 INFO:teuthology.orchestra.run.smithi105.stderr: 8 Mar 04:18:27 ntpd[9007]: can't open /var/log/ntpstats/loopstats.20220308: Permission denied 2022-03-08T04:18:27.903 INFO:teuthology.orchestra.run.smithi018.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T04:18:27.903 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================== 2022-03-08T04:18:27.903 INFO:teuthology.orchestra.run.smithi018.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.904 INFO:teuthology.orchestra.run.smithi018.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.904 INFO:teuthology.orchestra.run.smithi018.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.904 INFO:teuthology.orchestra.run.smithi018.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.913 INFO:teuthology.orchestra.run.smithi105.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T04:18:27.914 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================== 2022-03-08T04:18:27.914 INFO:teuthology.orchestra.run.smithi105.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.914 INFO:teuthology.orchestra.run.smithi105.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.915 INFO:teuthology.orchestra.run.smithi105.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.915 INFO:teuthology.orchestra.run.smithi105.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.916 INFO:teuthology.orchestra.run.smithi084.stdout: remote refid st t when poll reach delay offset jitter 2022-03-08T04:18:27.917 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================== 2022-03-08T04:18:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.917 INFO:teuthology.orchestra.run.smithi084.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-08T04:18:27.919 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-08T04:18:27.935 INFO:tasks.kubeadm:Kubeadm start 2022-03-08T04:18:27.936 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-08T04:18:27.936 DEBUG:teuthology.orchestra.run.smithi018:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T04:18:27.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T04:18:27.960 DEBUG:teuthology.orchestra.run.smithi105:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-08T04:18:27.980 INFO:teuthology.orchestra.run.smithi018.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T04:18:27.986 INFO:teuthology.orchestra.run.smithi084.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T04:18:27.987 INFO:teuthology.orchestra.run.smithi105.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-08T04:18:27.990 INFO:teuthology.orchestra.run.smithi084.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T04:18:27.991 INFO:teuthology.orchestra.run.smithi018.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T04:18:27.995 INFO:teuthology.orchestra.run.smithi105.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-08T04:18:27.997 INFO:teuthology.orchestra.run.smithi084.stdout:net.ipv4.ip_forward = 1 2022-03-08T04:18:28.000 INFO:teuthology.orchestra.run.smithi018.stdout:net.ipv4.ip_forward = 1 2022-03-08T04:18:28.001 INFO:teuthology.orchestra.run.smithi105.stdout:net.ipv4.ip_forward = 1 2022-03-08T04:18:28.058 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:18:28.059 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T04:18:28.075 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:18:28.075 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/docker/daemon.json 2022-03-08T04:18:28.134 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:18:28.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T04:18:28.149 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:18:28.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/docker/daemon.json 2022-03-08T04:18:28.163 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:18:28.163 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-08T04:18:28.179 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:18:28.179 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/docker/daemon.json 2022-03-08T04:18:28.190 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl restart docker || true 2022-03-08T04:18:28.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl restart docker || true 2022-03-08T04:18:28.196 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl restart docker || true 2022-03-08T04:18:29.297 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-08T04:18:29.305 INFO:tasks.kubeadm:172.21.15.18 is in 172.21.0.0/20 at pos 3857 2022-03-08T04:18:29.321 INFO:tasks.kubeadm:Pod subnet: 10.248.136.0/21 2022-03-08T04:18:29.323 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-08T04:18:29.323 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T04:18:29.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T04:18:29.329 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-08T04:18:29.350 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-08T04:18:29.351 INFO:teuthology.orchestra.run.smithi018.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:29.351 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-08T04:18:29.352 INFO:teuthology.orchestra.run.smithi084.stderr: 2022-03-08T04:18:29.352 INFO:teuthology.orchestra.run.smithi084.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:29.352 INFO:teuthology.orchestra.run.smithi084.stderr: 2022-03-08T04:18:29.359 INFO:teuthology.orchestra.run.smithi105.stderr: 2022-03-08T04:18:29.359 INFO:teuthology.orchestra.run.smithi105.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:29.360 INFO:teuthology.orchestra.run.smithi105.stderr: 2022-03-08T04:18:29.465 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T04:18:29.470 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T04:18:29.577 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:18:29.582 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:18:29.599 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T04:18:29.600 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:18:29.659 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T04:18:29.665 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T04:18:29.669 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T04:18:29.741 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T04:18:29.747 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T04:18:29.751 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T04:18:30.948 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2022-03-08T04:18:30.998 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-08T04:18:31.081 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2022-03-08T04:18:31.118 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2022-03-08T04:18:31.119 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2022-03-08T04:18:31.177 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-08T04:18:31.177 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-08T04:18:31.213 INFO:teuthology.orchestra.run.smithi084.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T04:18:31.226 INFO:teuthology.orchestra.run.smithi084.stderr: 2022-03-08T04:18:31.226 INFO:teuthology.orchestra.run.smithi084.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:31.227 INFO:teuthology.orchestra.run.smithi084.stderr: 2022-03-08T04:18:31.254 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2022-03-08T04:18:31.254 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2022-03-08T04:18:31.275 INFO:teuthology.orchestra.run.smithi018.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T04:18:31.280 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2022-03-08T04:18:31.290 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-08T04:18:31.291 INFO:teuthology.orchestra.run.smithi018.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:31.291 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-08T04:18:31.346 INFO:teuthology.orchestra.run.smithi105.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T04:18:31.349 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-08T04:18:31.360 INFO:teuthology.orchestra.run.smithi105.stderr: 2022-03-08T04:18:31.361 INFO:teuthology.orchestra.run.smithi105.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:31.361 INFO:teuthology.orchestra.run.smithi105.stderr: 2022-03-08T04:18:31.415 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2022-03-08T04:18:31.451 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2022-03-08T04:18:31.452 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2022-03-08T04:18:31.523 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-08T04:18:31.523 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-08T04:18:31.587 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2022-03-08T04:18:31.587 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2022-03-08T04:18:31.615 INFO:teuthology.orchestra.run.smithi084.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T04:18:31.616 INFO:teuthology.orchestra.run.smithi084.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T04:18:31.616 INFO:teuthology.orchestra.run.smithi084.stdout:The following NEW packages will be installed: 2022-03-08T04:18:31.617 INFO:teuthology.orchestra.run.smithi084.stdout: apt-transport-https 2022-03-08T04:18:31.717 INFO:teuthology.orchestra.run.smithi018.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T04:18:31.718 INFO:teuthology.orchestra.run.smithi018.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T04:18:31.718 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2022-03-08T04:18:31.718 INFO:teuthology.orchestra.run.smithi018.stdout: apt-transport-https 2022-03-08T04:18:31.760 INFO:teuthology.orchestra.run.smithi105.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-08T04:18:31.760 INFO:teuthology.orchestra.run.smithi105.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-08T04:18:31.761 INFO:teuthology.orchestra.run.smithi105.stdout:The following NEW packages will be installed: 2022-03-08T04:18:31.761 INFO:teuthology.orchestra.run.smithi105.stdout: apt-transport-https 2022-03-08T04:18:31.799 INFO:teuthology.orchestra.run.smithi084.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-08T04:18:31.799 INFO:teuthology.orchestra.run.smithi084.stdout:Need to get 4,680 B of archives. 2022-03-08T04:18:31.800 INFO:teuthology.orchestra.run.smithi084.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T04:18:31.800 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T04:18:31.907 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-08T04:18:31.908 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 4,680 B of archives. 2022-03-08T04:18:31.908 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T04:18:31.908 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T04:18:31.945 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-08T04:18:31.946 INFO:teuthology.orchestra.run.smithi105.stdout:Need to get 4,680 B of archives. 2022-03-08T04:18:31.946 INFO:teuthology.orchestra.run.smithi105.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-08T04:18:31.946 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-08T04:18:32.020 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T04:18:32.021 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T04:18:32.021 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: falling back to frontend: Readline 2022-03-08T04:18:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T04:18:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T04:18:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: falling back to frontend: Teletype 2022-03-08T04:18:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T04:18:32.084 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 4,680 B in 0s (26.6 kB/s) 2022-03-08T04:18:32.157 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T04:18:32.158 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T04:18:32.158 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: falling back to frontend: Readline 2022-03-08T04:18:32.160 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T04:18:32.161 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T04:18:32.161 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: falling back to frontend: Readline 2022-03-08T04:18:32.169 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T04:18:32.178 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T04:18:32.178 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T04:18:32.179 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: falling back to frontend: Teletype 2022-03-08T04:18:32.179 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T04:18:32.180 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T04:18:32.180 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: falling back to frontend: Teletype 2022-03-08T04:18:32.184 INFO:teuthology.orchestra.run.smithi105.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T04:18:32.185 INFO:teuthology.orchestra.run.smithi018.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T04:18:32.212 INFO:teuthology.orchestra.run.smithi084.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T04:18:32.215 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T04:18:32.232 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 4,680 B in 0s (26.3 kB/s) 2022-03-08T04:18:32.237 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 4,680 B in 0s (26.4 kB/s) 2022-03-08T04:18:32.260 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T04:18:32.306 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T04:18:32.322 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package apt-transport-https. 2022-03-08T04:18:32.350 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T04:18:32.353 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T04:18:32.362 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-08T04:18:32.365 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-08T04:18:32.396 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T04:18:32.404 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-08T04:18:32.555 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T04:18:32.694 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T04:18:32.708 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-08T04:18:33.158 INFO:teuthology.orchestra.run.smithi084.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T04:18:33.172 INFO:teuthology.orchestra.run.smithi084.stderr: 2022-03-08T04:18:33.172 INFO:teuthology.orchestra.run.smithi084.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:33.172 INFO:teuthology.orchestra.run.smithi084.stderr: 2022-03-08T04:18:33.270 INFO:teuthology.orchestra.run.smithi018.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T04:18:33.286 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-08T04:18:33.286 INFO:teuthology.orchestra.run.smithi018.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:33.287 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-08T04:18:33.302 INFO:teuthology.orchestra.run.smithi105.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-08T04:18:33.316 INFO:teuthology.orchestra.run.smithi105.stderr: 2022-03-08T04:18:33.316 INFO:teuthology.orchestra.run.smithi105.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:33.317 INFO:teuthology.orchestra.run.smithi105.stderr: 2022-03-08T04:18:33.484 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:18:33.484 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T04:18:33.485 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T04:18:33.549 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T04:18:33.564 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-08T04:18:33.565 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:18:33.565 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-08T04:18:33.632 INFO:teuthology.orchestra.run.smithi084.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T04:18:33.633 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T04:18:33.646 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-08T04:18:33.716 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T04:18:33.729 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-08T04:18:33.731 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T04:18:33.906 INFO:teuthology.orchestra.run.smithi105.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T04:18:33.934 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-08T04:18:34.114 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T04:18:34.129 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T04:18:34.218 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-08T04:18:34.247 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 63.3 kB in 1s (81.1 kB/s) 2022-03-08T04:18:34.268 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 63.3 kB in 1s (66.7 kB/s) 2022-03-08T04:18:34.365 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 63.3 kB in 1s (68.5 kB/s) 2022-03-08T04:18:35.081 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2022-03-08T04:18:35.115 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2022-03-08T04:18:35.245 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-08T04:18:35.251 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2022-03-08T04:18:35.252 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2022-03-08T04:18:35.286 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2022-03-08T04:18:35.286 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2022-03-08T04:18:35.342 INFO:teuthology.orchestra.run.smithi105.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T04:18:35.351 INFO:teuthology.orchestra.run.smithi105.stderr: 2022-03-08T04:18:35.351 INFO:teuthology.orchestra.run.smithi105.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:35.351 INFO:teuthology.orchestra.run.smithi105.stderr: 2022-03-08T04:18:35.378 INFO:teuthology.orchestra.run.smithi084.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T04:18:35.389 INFO:teuthology.orchestra.run.smithi084.stderr: 2022-03-08T04:18:35.390 INFO:teuthology.orchestra.run.smithi084.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:35.390 INFO:teuthology.orchestra.run.smithi084.stderr: 2022-03-08T04:18:35.409 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2022-03-08T04:18:35.422 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-08T04:18:35.422 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-08T04:18:35.447 INFO:teuthology.orchestra.run.smithi084.stdout:Reading package lists... 2022-03-08T04:18:35.511 INFO:teuthology.orchestra.run.smithi018.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-08T04:18:35.525 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-08T04:18:35.525 INFO:teuthology.orchestra.run.smithi018.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-08T04:18:35.525 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-08T04:18:35.563 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2022-03-08T04:18:35.564 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2022-03-08T04:18:35.582 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-08T04:18:35.604 INFO:teuthology.orchestra.run.smithi084.stdout:Building dependency tree... 2022-03-08T04:18:35.605 INFO:teuthology.orchestra.run.smithi084.stdout:Reading state information... 2022-03-08T04:18:35.693 INFO:teuthology.orchestra.run.smithi105.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T04:18:35.693 INFO:teuthology.orchestra.run.smithi105.stdout:bridge-utils set to manually installed. 2022-03-08T04:18:35.694 INFO:teuthology.orchestra.run.smithi105.stdout:The following additional packages will be installed: 2022-03-08T04:18:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T04:18:35.694 INFO:teuthology.orchestra.run.smithi105.stdout:Suggested packages: 2022-03-08T04:18:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: nftables 2022-03-08T04:18:35.733 INFO:teuthology.orchestra.run.smithi084.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T04:18:35.734 INFO:teuthology.orchestra.run.smithi084.stdout:bridge-utils set to manually installed. 2022-03-08T04:18:35.734 INFO:teuthology.orchestra.run.smithi084.stdout:The following additional packages will be installed: 2022-03-08T04:18:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T04:18:35.735 INFO:teuthology.orchestra.run.smithi084.stdout:Suggested packages: 2022-03-08T04:18:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: nftables 2022-03-08T04:18:35.738 INFO:teuthology.orchestra.run.smithi105.stdout:The following NEW packages will be installed: 2022-03-08T04:18:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T04:18:35.745 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-08T04:18:35.745 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-08T04:18:35.780 INFO:teuthology.orchestra.run.smithi084.stdout:The following NEW packages will be installed: 2022-03-08T04:18:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T04:18:35.884 INFO:teuthology.orchestra.run.smithi018.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-08T04:18:35.884 INFO:teuthology.orchestra.run.smithi018.stdout:bridge-utils set to manually installed. 2022-03-08T04:18:35.884 INFO:teuthology.orchestra.run.smithi018.stdout:The following additional packages will be installed: 2022-03-08T04:18:35.885 INFO:teuthology.orchestra.run.smithi018.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-08T04:18:35.885 INFO:teuthology.orchestra.run.smithi018.stdout:Suggested packages: 2022-03-08T04:18:35.886 INFO:teuthology.orchestra.run.smithi018.stdout: nftables 2022-03-08T04:18:35.931 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-08T04:18:35.931 INFO:teuthology.orchestra.run.smithi105.stdout:Need to get 73.6 MB of archives. 2022-03-08T04:18:35.932 INFO:teuthology.orchestra.run.smithi105.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T04:18:35.932 INFO:teuthology.orchestra.run.smithi105.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T04:18:35.938 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2022-03-08T04:18:35.940 INFO:teuthology.orchestra.run.smithi018.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-08T04:18:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T04:18:35.973 INFO:teuthology.orchestra.run.smithi084.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-08T04:18:35.973 INFO:teuthology.orchestra.run.smithi084.stdout:Need to get 73.6 MB of archives. 2022-03-08T04:18:35.973 INFO:teuthology.orchestra.run.smithi084.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T04:18:35.974 INFO:teuthology.orchestra.run.smithi084.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T04:18:36.015 INFO:teuthology.orchestra.run.smithi084.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T04:18:36.160 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-08T04:18:36.161 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 73.6 MB of archives. 2022-03-08T04:18:36.161 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-08T04:18:36.161 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-08T04:18:36.173 INFO:teuthology.orchestra.run.smithi084.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T04:18:36.209 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T04:18:36.227 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-08T04:18:36.232 INFO:teuthology.orchestra.run.smithi105.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T04:18:36.294 INFO:teuthology.orchestra.run.smithi084.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T04:18:36.313 INFO:teuthology.orchestra.run.smithi084.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T04:18:36.341 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-08T04:18:36.344 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T04:18:36.486 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-08T04:18:36.809 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-08T04:18:36.900 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T04:18:36.941 INFO:teuthology.orchestra.run.smithi084.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T04:18:37.369 INFO:teuthology.orchestra.run.smithi084.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T04:18:37.496 INFO:teuthology.orchestra.run.smithi084.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T04:18:37.498 INFO:teuthology.orchestra.run.smithi105.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T04:18:37.598 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-08T04:18:37.685 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T04:18:37.885 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T04:18:37.886 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T04:18:37.886 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: falling back to frontend: Readline 2022-03-08T04:18:37.893 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T04:18:37.893 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T04:18:37.893 INFO:teuthology.orchestra.run.smithi084.stderr:debconf: falling back to frontend: Teletype 2022-03-08T04:18:37.898 INFO:teuthology.orchestra.run.smithi084.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T04:18:37.948 INFO:teuthology.orchestra.run.smithi084.stdout:Fetched 73.6 MB in 2s (39.2 MB/s) 2022-03-08T04:18:37.991 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-08T04:18:38.033 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package conntrack. 2022-03-08T04:18:38.070 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T04:18:38.071 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T04:18:38.071 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: falling back to frontend: Readline 2022-03-08T04:18:38.075 INFO:teuthology.orchestra.run.smithi084.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T04:18:38.078 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T04:18:38.078 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T04:18:38.079 INFO:teuthology.orchestra.run.smithi105.stderr:debconf: falling back to frontend: Teletype 2022-03-08T04:18:38.079 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T04:18:38.084 INFO:teuthology.orchestra.run.smithi105.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T04:18:38.105 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-08T04:18:38.123 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T04:18:38.134 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 73.6 MB in 2s (34.9 MB/s) 2022-03-08T04:18:38.195 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package conntrack. 2022-03-08T04:18:38.237 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T04:18:38.240 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T04:18:38.269 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T04:18:38.376 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package cri-tools. 2022-03-08T04:18:38.390 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T04:18:38.426 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T04:18:38.477 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: unable to initialize frontend: Dialog 2022-03-08T04:18:38.478 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-08T04:18:38.478 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: falling back to frontend: Readline 2022-03-08T04:18:38.487 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: unable to initialize frontend: Readline 2022-03-08T04:18:38.488 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-08T04:18:38.488 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: falling back to frontend: Teletype 2022-03-08T04:18:38.493 INFO:teuthology.orchestra.run.smithi018.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-08T04:18:38.541 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 73.6 MB in 2s (31.9 MB/s) 2022-03-08T04:18:38.613 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package cri-tools. 2022-03-08T04:18:38.622 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package conntrack. 2022-03-08T04:18:38.625 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T04:18:38.666 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-08T04:18:38.670 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-08T04:18:38.671 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T04:18:38.712 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-08T04:18:39.040 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package cri-tools. 2022-03-08T04:18:39.056 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-08T04:18:39.099 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-08T04:18:39.528 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package ebtables. 2022-03-08T04:18:39.541 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T04:18:39.594 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T04:18:39.791 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ebtables. 2022-03-08T04:18:39.805 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T04:18:39.857 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T04:18:39.857 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T04:18:39.870 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T04:18:39.914 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T04:18:40.144 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T04:18:40.158 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T04:18:40.194 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T04:18:40.218 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ebtables. 2022-03-08T04:18:40.233 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-08T04:18:40.313 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-08T04:18:40.546 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-08T04:18:40.560 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-08T04:18:40.617 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-08T04:18:41.400 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package socat. 2022-03-08T04:18:41.413 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T04:18:41.458 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T04:18:41.621 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package socat. 2022-03-08T04:18:41.634 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T04:18:41.688 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T04:18:41.828 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package kubelet. 2022-03-08T04:18:41.841 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T04:18:41.869 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T04:18:42.101 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package kubelet. 2022-03-08T04:18:42.101 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package socat. 2022-03-08T04:18:42.110 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T04:18:42.112 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-08T04:18:42.140 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-08T04:18:42.145 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T04:18:42.509 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package kubelet. 2022-03-08T04:18:42.517 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-08T04:18:42.569 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-08T04:18:44.276 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package kubectl. 2022-03-08T04:18:44.284 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T04:18:44.324 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T04:18:44.597 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package kubectl. 2022-03-08T04:18:44.611 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T04:18:44.681 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T04:18:45.028 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package kubectl. 2022-03-08T04:18:45.043 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-08T04:18:45.078 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-08T04:18:45.535 INFO:teuthology.orchestra.run.smithi084.stdout:Selecting previously unselected package kubeadm. 2022-03-08T04:18:45.547 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T04:18:45.584 INFO:teuthology.orchestra.run.smithi084.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T04:18:45.875 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package kubeadm. 2022-03-08T04:18:45.888 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T04:18:45.949 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T04:18:46.288 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package kubeadm. 2022-03-08T04:18:46.302 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-08T04:18:46.371 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-08T04:18:46.737 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T04:18:46.865 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T04:18:47.025 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T04:18:47.131 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T04:18:47.289 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T04:18:47.415 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T04:18:47.502 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-08T04:18:47.645 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-08T04:18:47.765 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-08T04:18:47.865 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T04:18:48.000 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T04:18:48.127 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T04:18:48.253 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T04:18:48.264 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T04:18:48.389 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T04:18:48.475 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T04:18:48.533 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T04:18:48.642 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T04:18:48.644 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-08T04:18:48.771 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-08T04:18:48.850 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T04:18:48.896 INFO:teuthology.orchestra.run.smithi084.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T04:18:48.921 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-08T04:18:49.048 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-08T04:18:49.275 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-08T04:18:49.335 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T04:18:49.566 INFO:teuthology.orchestra.run.smithi084.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T04:18:49.766 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-08T04:18:49.879 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T04:18:50.260 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-08T04:18:51.084 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T04:18:51.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T04:18:51.131 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-08T04:19:03.910 INFO:teuthology.orchestra.run.smithi084.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T04:19:04.912 INFO:teuthology.orchestra.run.smithi105.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T04:19:05.703 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-08T04:19:07.062 INFO:teuthology.orchestra.run.smithi084.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T04:19:08.087 INFO:teuthology.orchestra.run.smithi105.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T04:19:08.732 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-08T04:19:09.448 INFO:teuthology.orchestra.run.smithi084.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T04:19:10.779 INFO:teuthology.orchestra.run.smithi105.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T04:19:10.913 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-08T04:19:12.975 INFO:teuthology.orchestra.run.smithi084.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T04:19:14.440 INFO:teuthology.orchestra.run.smithi084.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T04:19:14.445 INFO:teuthology.orchestra.run.smithi105.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T04:19:14.920 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-08T04:19:16.043 INFO:teuthology.orchestra.run.smithi105.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T04:19:16.578 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-08T04:19:19.589 INFO:teuthology.orchestra.run.smithi084.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T04:19:21.981 INFO:teuthology.orchestra.run.smithi084.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T04:19:22.127 INFO:teuthology.orchestra.run.smithi105.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T04:19:22.319 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-08T04:19:24.168 INFO:teuthology.orchestra.run.smithi105.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T04:19:24.485 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-08T04:19:24.490 INFO:tasks.kubeadm:Token: abcdef.ons9645mpzc55svo 2022-03-08T04:19:24.490 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): '172.21.15.18', Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): '172.21.15.84', Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): '172.21.15.105'} 2022-03-08T04:19:24.491 DEBUG:teuthology.orchestra.run.smithi018:> sudo kubeadm init --node-name smithi018 --token abcdef.ons9645mpzc55svo --pod-network-cidr 10.248.136.0/21 2022-03-08T04:19:24.691 INFO:teuthology.orchestra.run.smithi018.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-08T04:19:24.691 INFO:teuthology.orchestra.run.smithi018.stdout:[preflight] Running pre-flight checks 2022-03-08T04:19:25.119 INFO:teuthology.orchestra.run.smithi018.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-08T04:19:25.120 INFO:teuthology.orchestra.run.smithi018.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-08T04:19:25.120 INFO:teuthology.orchestra.run.smithi018.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-08T04:19:25.428 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-08T04:19:25.718 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "ca" certificate and key 2022-03-08T04:19:25.874 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "apiserver" certificate and key 2022-03-08T04:19:25.875 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi018] and IPs [10.96.0.1 172.21.15.18] 2022-03-08T04:19:26.061 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-08T04:19:26.389 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-08T04:19:26.503 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-08T04:19:26.663 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-08T04:19:27.014 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-08T04:19:27.014 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi018] and IPs [172.21.15.18 127.0.0.1 ::1] 2022-03-08T04:19:27.186 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-08T04:19:27.187 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi018] and IPs [172.21.15.18 127.0.0.1 ::1] 2022-03-08T04:19:27.302 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-08T04:19:27.365 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-08T04:19:27.492 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "sa" key and public key 2022-03-08T04:19:27.492 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-08T04:19:27.691 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-08T04:19:27.817 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-08T04:19:28.013 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-08T04:19:28.234 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-08T04:19:28.249 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T04:19:28.251 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T04:19:28.251 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet-start] Starting the kubelet 2022-03-08T04:19:28.469 INFO:teuthology.orchestra.run.smithi018.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-08T04:19:28.470 INFO:teuthology.orchestra.run.smithi018.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-08T04:19:28.471 INFO:teuthology.orchestra.run.smithi018.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-08T04:19:28.472 INFO:teuthology.orchestra.run.smithi018.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-08T04:19:28.473 INFO:teuthology.orchestra.run.smithi018.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-08T04:19:28.474 INFO:teuthology.orchestra.run.smithi018.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-08T04:19:46.479 INFO:teuthology.orchestra.run.smithi018.stdout:[apiclient] All control plane components are healthy after 18.003523 seconds 2022-03-08T04:19:46.479 INFO:teuthology.orchestra.run.smithi018.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-08T04:19:46.498 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-08T04:19:46.498 INFO:teuthology.orchestra.run.smithi018.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-08T04:19:47.157 INFO:teuthology.orchestra.run.smithi018.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-08T04:19:47.158 INFO:teuthology.orchestra.run.smithi018.stdout:[mark-control-plane] Marking the node smithi018 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-08T04:19:47.158 INFO:teuthology.orchestra.run.smithi018.stdout:[mark-control-plane] Marking the node smithi018 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-08T04:19:47.667 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] Using token: abcdef.ons9645mpzc55svo 2022-03-08T04:19:47.668 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-08T04:19:47.688 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-08T04:19:47.706 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-08T04:19:47.714 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-08T04:19:47.777 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-08T04:19:47.781 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-08T04:19:47.814 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-08T04:19:48.306 INFO:teuthology.orchestra.run.smithi018.stdout:[addons] Applied essential addon: CoreDNS 2022-03-08T04:19:48.449 INFO:teuthology.orchestra.run.smithi018.stdout:[addons] Applied essential addon: kube-proxy 2022-03-08T04:19:48.451 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:19:48.451 INFO:teuthology.orchestra.run.smithi018.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-08T04:19:48.451 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:19:48.451 INFO:teuthology.orchestra.run.smithi018.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-08T04:19:48.452 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:19:48.452 INFO:teuthology.orchestra.run.smithi018.stdout: mkdir -p $HOME/.kube 2022-03-08T04:19:48.452 INFO:teuthology.orchestra.run.smithi018.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-08T04:19:48.452 INFO:teuthology.orchestra.run.smithi018.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-08T04:19:48.453 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:19:48.453 INFO:teuthology.orchestra.run.smithi018.stdout:Alternatively, if you are the root user, you can run: 2022-03-08T04:19:48.453 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:19:48.453 INFO:teuthology.orchestra.run.smithi018.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-08T04:19:48.453 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:19:48.454 INFO:teuthology.orchestra.run.smithi018.stdout:You should now deploy a pod network to the cluster. 2022-03-08T04:19:48.454 INFO:teuthology.orchestra.run.smithi018.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-08T04:19:48.454 INFO:teuthology.orchestra.run.smithi018.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-08T04:19:48.454 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:19:48.455 INFO:teuthology.orchestra.run.smithi018.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-08T04:19:48.455 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:19:48.455 INFO:teuthology.orchestra.run.smithi018.stdout:kubeadm join 172.21.15.18:6443 --token abcdef.ons9645mpzc55svo \ 2022-03-08T04:19:48.455 INFO:teuthology.orchestra.run.smithi018.stdout: --discovery-token-ca-cert-hash sha256:19d893bb61cc6cd21efcc4c062d1f2878012af3583f8d56c113629e9918a0e4f 2022-03-08T04:19:48.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo kubeadm join 172.21.15.18:6443 --node-name smithi084 --token abcdef.ons9645mpzc55svo --discovery-token-unsafe-skip-ca-verification 2022-03-08T04:19:48.459 DEBUG:teuthology.orchestra.run.smithi105:> sudo kubeadm join 172.21.15.18:6443 --node-name smithi105 --token abcdef.ons9645mpzc55svo --discovery-token-unsafe-skip-ca-verification 2022-03-08T04:19:48.500 INFO:teuthology.orchestra.run.smithi105.stdout:[preflight] Running pre-flight checks 2022-03-08T04:19:48.500 INFO:teuthology.orchestra.run.smithi084.stdout:[preflight] Running pre-flight checks 2022-03-08T04:20:01.472 INFO:teuthology.orchestra.run.smithi105.stdout:[preflight] Reading configuration from the cluster... 2022-03-08T04:20:01.473 INFO:teuthology.orchestra.run.smithi105.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-08T04:20:01.940 INFO:teuthology.orchestra.run.smithi105.stderr:W0308 04:20:01.942018 10656 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-08T04:20:01.957 INFO:teuthology.orchestra.run.smithi105.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T04:20:01.957 INFO:teuthology.orchestra.run.smithi105.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T04:20:01.957 INFO:teuthology.orchestra.run.smithi105.stdout:[kubelet-start] Starting the kubelet 2022-03-08T04:20:02.184 INFO:teuthology.orchestra.run.smithi105.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-08T04:20:07.310 INFO:teuthology.orchestra.run.smithi084.stdout:[preflight] Reading configuration from the cluster... 2022-03-08T04:20:07.311 INFO:teuthology.orchestra.run.smithi084.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-08T04:20:07.663 INFO:teuthology.orchestra.run.smithi084.stderr:W0308 04:20:07.664826 10628 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-08T04:20:07.682 INFO:teuthology.orchestra.run.smithi084.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-08T04:20:07.682 INFO:teuthology.orchestra.run.smithi084.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-08T04:20:07.682 INFO:teuthology.orchestra.run.smithi084.stdout:[kubelet-start] Starting the kubelet 2022-03-08T04:20:07.889 INFO:teuthology.orchestra.run.smithi084.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-08T04:20:13.420 INFO:teuthology.orchestra.run.smithi084.stdout: 2022-03-08T04:20:13.421 INFO:teuthology.orchestra.run.smithi084.stdout:This node has joined the cluster: 2022-03-08T04:20:13.421 INFO:teuthology.orchestra.run.smithi084.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-08T04:20:13.421 INFO:teuthology.orchestra.run.smithi084.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-08T04:20:13.422 INFO:teuthology.orchestra.run.smithi084.stdout: 2022-03-08T04:20:13.422 INFO:teuthology.orchestra.run.smithi084.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-08T04:20:13.422 INFO:teuthology.orchestra.run.smithi084.stdout: 2022-03-08T04:20:15.750 INFO:teuthology.orchestra.run.smithi105.stdout: 2022-03-08T04:20:15.750 INFO:teuthology.orchestra.run.smithi105.stdout:This node has joined the cluster: 2022-03-08T04:20:15.750 INFO:teuthology.orchestra.run.smithi105.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-08T04:20:15.751 INFO:teuthology.orchestra.run.smithi105.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-08T04:20:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: 2022-03-08T04:20:15.751 INFO:teuthology.orchestra.run.smithi105.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-08T04:20:15.751 INFO:teuthology.orchestra.run.smithi105.stdout: 2022-03-08T04:20:15.756 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:20:15.757 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-08T04:20:15.771 INFO:tasks.kubeadm:Setting up kubectl 2022-03-08T04:20:15.772 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T04:20:15.833 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T04:20:15.850 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-08T04:20:15.868 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:20:15.869 DEBUG:teuthology.orchestra.run.smithi018:> dd of=.kube/config 2022-03-08T04:20:15.875 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:20:15.876 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/root/.kube/config 2022-03-08T04:20:15.888 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:20:15.889 DEBUG:teuthology.orchestra.run.smithi084:> dd of=.kube/config 2022-03-08T04:20:15.905 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:20:15.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/root/.kube/config 2022-03-08T04:20:15.918 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:20:15.919 DEBUG:teuthology.orchestra.run.smithi105:> dd of=.kube/config 2022-03-08T04:20:15.927 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:20:15.928 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/root/.kube/config 2022-03-08T04:20:15.987 DEBUG:teuthology.orchestra.run.smithi018:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-08T04:20:18.871 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-08T04:20:18.920 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-08T04:20:19.015 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-08T04:20:19.089 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-08T04:20:19.221 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-08T04:20:19.260 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-08T04:20:19.315 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-08T04:20:19.412 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-08T04:20:19.454 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-08T04:20:19.579 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-08T04:20:19.586 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-08T04:20:19.638 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-08T04:20:19.716 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-08T04:20:19.754 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-08T04:20:19.883 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-08T04:20:19.948 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-08T04:20:19.980 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-08T04:20:20.072 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-08T04:20:20.106 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-08T04:20:20.145 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-08T04:20:20.262 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-08T04:20:20.344 INFO:teuthology.orchestra.run.smithi018.stdout:namespace/tigera-operator created 2022-03-08T04:20:20.423 INFO:teuthology.orchestra.run.smithi018.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T04:20:20.429 INFO:teuthology.orchestra.run.smithi018.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-08T04:20:20.460 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/tigera-operator created 2022-03-08T04:20:20.498 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T04:20:20.518 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-08T04:20:20.571 INFO:teuthology.orchestra.run.smithi018.stdout:deployment.apps/tigera-operator created 2022-03-08T04:20:20.577 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f - 2022-03-08T04:20:23.509 INFO:teuthology.orchestra.run.smithi018.stdout:installation.operator.tigera.io/default created 2022-03-08T04:20:23.517 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-08T04:20:23.517 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:20:23.518 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:20:23.524 INFO:tasks.kubeadm:Scratch devices on smithi018: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T04:20:23.525 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T04:20:23.594 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records in 2022-03-08T04:20:23.595 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records out 2022-03-08T04:20:23.595 INFO:teuthology.orchestra.run.smithi018.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150409 s, 697 MB/s 2022-03-08T04:20:23.597 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T04:20:23.625 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records in 2022-03-08T04:20:23.626 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records out 2022-03-08T04:20:23.626 INFO:teuthology.orchestra.run.smithi018.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0146263 s, 717 MB/s 2022-03-08T04:20:23.627 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T04:20:23.696 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records in 2022-03-08T04:20:23.697 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records out 2022-03-08T04:20:23.697 INFO:teuthology.orchestra.run.smithi018.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140659 s, 745 MB/s 2022-03-08T04:20:23.698 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T04:20:23.719 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records in 2022-03-08T04:20:23.719 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records out 2022-03-08T04:20:23.719 INFO:teuthology.orchestra.run.smithi018.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116217 s, 902 MB/s 2022-03-08T04:20:23.721 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:20:23.721 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:20:23.726 INFO:tasks.kubeadm:Scratch devices on smithi084: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T04:20:23.727 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T04:20:23.751 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records in 2022-03-08T04:20:23.751 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2022-03-08T04:20:23.752 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124713 s, 841 MB/s 2022-03-08T04:20:23.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T04:20:23.824 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records in 2022-03-08T04:20:23.824 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2022-03-08T04:20:23.825 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0171774 s, 610 MB/s 2022-03-08T04:20:23.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T04:20:23.858 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records in 2022-03-08T04:20:23.858 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2022-03-08T04:20:23.859 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163983 s, 639 MB/s 2022-03-08T04:20:23.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T04:20:23.936 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records in 2022-03-08T04:20:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2022-03-08T04:20:23.937 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0165129 s, 635 MB/s 2022-03-08T04:20:23.938 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:20:23.939 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:20:23.947 INFO:tasks.kubeadm:Scratch devices on smithi105: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T04:20:23.947 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-08T04:20:23.982 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records in 2022-03-08T04:20:23.983 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2022-03-08T04:20:23.984 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0184009 s, 570 MB/s 2022-03-08T04:20:23.986 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-08T04:20:24.054 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records in 2022-03-08T04:20:24.054 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2022-03-08T04:20:24.055 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0164343 s, 638 MB/s 2022-03-08T04:20:24.055 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-08T04:20:24.081 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records in 2022-03-08T04:20:24.081 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2022-03-08T04:20:24.082 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0146787 s, 714 MB/s 2022-03-08T04:20:24.083 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-08T04:20:24.113 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records in 2022-03-08T04:20:24.114 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2022-03-08T04:20:24.114 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0168457 s, 622 MB/s 2022-03-08T04:20:24.136 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-08T04:20:24.137 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi018-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi018 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi018-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi018 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi018-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi018 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi018-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi018 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi084-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi084 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi084-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi084 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi084-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi084 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi084-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi084 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi105-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi105 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi105-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi105 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi105-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi105 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi105-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi105 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-08T04:20:24.137 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f - 2022-03-08T04:20:25.020 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi018-lv-1 created 2022-03-08T04:20:25.026 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi018-lv-2 created 2022-03-08T04:20:25.035 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi018-lv-3 created 2022-03-08T04:20:25.103 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi018-lv-4 created 2022-03-08T04:20:25.106 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi084-lv-1 created 2022-03-08T04:20:25.118 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi084-lv-2 created 2022-03-08T04:20:25.127 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi084-lv-3 created 2022-03-08T04:20:25.160 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi084-lv-4 created 2022-03-08T04:20:25.168 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi105-lv-1 created 2022-03-08T04:20:25.194 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi105-lv-2 created 2022-03-08T04:20:25.201 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi105-lv-3 created 2022-03-08T04:20:25.255 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi105-lv-4 created 2022-03-08T04:20:25.258 INFO:teuthology.orchestra.run.smithi018.stdout:storageclass.storage.k8s.io/scratch created 2022-03-08T04:20:25.263 DEBUG:teuthology.orchestra.run.smithi018:> kubectl taint node smithi018 node-role.kubernetes.io/master- || true 2022-03-08T04:20:25.337 INFO:teuthology.orchestra.run.smithi018.stdout:node/smithi018 untainted 2022-03-08T04:20:25.339 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-08T04:20:25.339 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-08T04:20:25.366 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-08T04:20:25.367 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:20:25.368 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:20:25.377 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T04:20:25.378 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T04:20:25.384 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T04:20:25.385 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:25.385 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-08T04:20:25.385 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:25.385 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2022-03-08 04:20:23.594937910 +0000 2022-03-08T04:20:25.386 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2022-03-08 04:20:23.594937910 +0000 2022-03-08T04:20:25.386 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2022-03-08 04:20:23.594937910 +0000 2022-03-08T04:20:25.386 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2022-03-08T04:20:25.393 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2022-03-08T04:20:25.393 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2022-03-08T04:20:25.393 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000175097 s, 2.9 MB/s 2022-03-08T04:20:25.398 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T04:20:25.451 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T04:20:25.451 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:25.451 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-08T04:20:25.452 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:25.452 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2022-03-08 04:20:23.626937909 +0000 2022-03-08T04:20:25.452 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2022-03-08 04:20:23.626937909 +0000 2022-03-08T04:20:25.452 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2022-03-08 04:20:23.626937909 +0000 2022-03-08T04:20:25.453 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2022-03-08T04:20:25.460 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2022-03-08T04:20:25.460 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2022-03-08T04:20:25.460 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000343581 s, 1.5 MB/s 2022-03-08T04:20:25.465 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T04:20:25.472 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T04:20:25.472 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:25.472 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-03-08T04:20:25.472 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:25.473 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2022-03-08 04:20:23.694937908 +0000 2022-03-08T04:20:25.473 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2022-03-08 04:20:23.694937908 +0000 2022-03-08T04:20:25.473 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2022-03-08 04:20:23.694937908 +0000 2022-03-08T04:20:25.473 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2022-03-08T04:20:25.479 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2022-03-08T04:20:25.479 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2022-03-08T04:20:25.480 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000312455 s, 1.6 MB/s 2022-03-08T04:20:25.484 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T04:20:25.491 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T04:20:25.491 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:25.491 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-08T04:20:25.492 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:25.492 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2022-03-08 04:20:23.718937907 +0000 2022-03-08T04:20:25.492 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2022-03-08 04:20:23.718937907 +0000 2022-03-08T04:20:25.492 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2022-03-08 04:20:23.718937907 +0000 2022-03-08T04:20:25.493 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2022-03-08T04:20:25.499 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2022-03-08T04:20:25.499 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2022-03-08T04:20:25.499 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000166487 s, 3.1 MB/s 2022-03-08T04:20:25.504 DEBUG:teuthology.orchestra.run.smithi018:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T04:20:25.591 INFO:teuthology.orchestra.run.smithi018.stdout:loop 2022-03-08T04:20:25.593 INFO:tasks.nvme_loop:Connecting nvme_loop smithi018:/dev/vg_nvme/lv_1... 2022-03-08T04:20:25.593 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T04:20:25.619 INFO:teuthology.orchestra.run.smithi018.stdout:1 2022-03-08T04:20:25.645 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_11 2022-03-08T04:20:25.669 INFO:tasks.nvme_loop:Connecting nvme_loop smithi018:/dev/vg_nvme/lv_2... 2022-03-08T04:20:25.669 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T04:20:25.736 INFO:teuthology.orchestra.run.smithi018.stdout:1 2022-03-08T04:20:25.758 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_21 2022-03-08T04:20:25.778 INFO:tasks.nvme_loop:Connecting nvme_loop smithi018:/dev/vg_nvme/lv_3... 2022-03-08T04:20:25.779 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T04:20:25.798 INFO:teuthology.orchestra.run.smithi018.stdout:1 2022-03-08T04:20:25.826 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_31 2022-03-08T04:20:25.848 INFO:tasks.nvme_loop:Connecting nvme_loop smithi018:/dev/vg_nvme/lv_4... 2022-03-08T04:20:25.849 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T04:20:25.875 INFO:teuthology.orchestra.run.smithi018.stdout:1 2022-03-08T04:20:25.892 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_41 2022-03-08T04:20:25.916 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:20:25.916 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:20:25.964 DEBUG:teuthology.orchestra.run.smithi018:> sudo nvme list 2022-03-08T04:20:25.978 INFO:teuthology.orchestra.run.smithi018.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T04:20:25.978 INFO:teuthology.orchestra.run.smithi018.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T04:20:25.978 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme0n1 PHMB75120091480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-08T04:20:25.978 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme1n1 fb8bdf65028d9631 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:25.979 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme2n1 79a06ca4798147f5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:25.979 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme3n1 aa02095e20a7bdfc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:25.979 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme4n1 798f9423b66bd5a2 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:25.980 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T04:20:25.981 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:20:25.981 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/scratch_devs 2022-03-08T04:20:25.998 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:20:25.998 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:20:26.006 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T04:20:26.006 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T04:20:26.013 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T04:20:26.013 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:26.014 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-08T04:20:26.014 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:26.014 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2022-03-08 04:20:23.753590291 +0000 2022-03-08T04:20:26.014 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2022-03-08 04:20:23.749590291 +0000 2022-03-08T04:20:26.015 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2022-03-08 04:20:23.749590291 +0000 2022-03-08T04:20:26.015 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2022-03-08T04:20:26.024 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2022-03-08T04:20:26.025 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2022-03-08T04:20:26.025 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000597033 s, 858 kB/s 2022-03-08T04:20:26.029 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T04:20:26.077 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T04:20:26.077 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:26.077 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-03-08T04:20:26.078 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:26.078 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2022-03-08 04:20:23.825590293 +0000 2022-03-08T04:20:26.078 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2022-03-08 04:20:23.825590293 +0000 2022-03-08T04:20:26.078 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2022-03-08 04:20:23.825590293 +0000 2022-03-08T04:20:26.079 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2022-03-08T04:20:26.085 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2022-03-08T04:20:26.086 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2022-03-08T04:20:26.086 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000462441 s, 1.1 MB/s 2022-03-08T04:20:26.090 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T04:20:26.097 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T04:20:26.097 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:26.098 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-03-08T04:20:26.098 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:26.098 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2022-03-08 04:20:23.861590294 +0000 2022-03-08T04:20:26.099 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2022-03-08 04:20:23.861590294 +0000 2022-03-08T04:20:26.099 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2022-03-08 04:20:23.861590294 +0000 2022-03-08T04:20:26.099 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2022-03-08T04:20:26.105 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2022-03-08T04:20:26.106 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2022-03-08T04:20:26.106 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000480914 s, 1.1 MB/s 2022-03-08T04:20:26.110 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T04:20:26.116 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T04:20:26.116 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:26.117 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-03-08T04:20:26.117 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:26.117 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2022-03-08 04:20:23.937590296 +0000 2022-03-08T04:20:26.117 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2022-03-08 04:20:23.937590296 +0000 2022-03-08T04:20:26.118 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2022-03-08 04:20:23.937590296 +0000 2022-03-08T04:20:26.118 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2022-03-08T04:20:26.124 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2022-03-08T04:20:26.124 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2022-03-08T04:20:26.124 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000554238 s, 924 kB/s 2022-03-08T04:20:26.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T04:20:26.256 INFO:teuthology.orchestra.run.smithi084.stdout:loop 2022-03-08T04:20:26.258 INFO:tasks.nvme_loop:Connecting nvme_loop smithi084:/dev/vg_nvme/lv_1... 2022-03-08T04:20:26.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T04:20:26.279 INFO:teuthology.orchestra.run.smithi084.stdout:1 2022-03-08T04:20:26.303 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/vg_nvme/lv_11 2022-03-08T04:20:26.326 INFO:tasks.nvme_loop:Connecting nvme_loop smithi084:/dev/vg_nvme/lv_2... 2022-03-08T04:20:26.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T04:20:26.398 INFO:teuthology.orchestra.run.smithi084.stdout:1 2022-03-08T04:20:26.427 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/vg_nvme/lv_21 2022-03-08T04:20:26.449 INFO:tasks.nvme_loop:Connecting nvme_loop smithi084:/dev/vg_nvme/lv_3... 2022-03-08T04:20:26.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T04:20:26.470 INFO:teuthology.orchestra.run.smithi084.stdout:1 2022-03-08T04:20:26.489 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/vg_nvme/lv_31 2022-03-08T04:20:26.510 INFO:tasks.nvme_loop:Connecting nvme_loop smithi084:/dev/vg_nvme/lv_4... 2022-03-08T04:20:26.511 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T04:20:26.532 INFO:teuthology.orchestra.run.smithi084.stdout:1 2022-03-08T04:20:26.553 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/vg_nvme/lv_41 2022-03-08T04:20:26.575 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:20:26.575 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:20:26.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo nvme list 2022-03-08T04:20:26.635 INFO:teuthology.orchestra.run.smithi084.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T04:20:26.635 INFO:teuthology.orchestra.run.smithi084.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T04:20:26.635 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme0n1 PHFT620400WF400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T04:20:26.636 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme1n1 6d7569888a77c40b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:26.636 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme2n1 88ca99da20a2b875 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:26.636 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme3n1 f48b93b389643635 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:26.636 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme4n1 840a637a8548baea Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:26.637 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T04:20:26.638 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:20:26.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/scratch_devs 2022-03-08T04:20:26.649 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:20:26.650 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:20:26.655 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-08T04:20:26.656 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-08T04:20:26.661 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-08T04:20:26.661 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:26.662 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-08T04:20:26.662 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:26.662 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2022-03-08 04:20:23.985649081 +0000 2022-03-08T04:20:26.662 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2022-03-08 04:20:23.985649081 +0000 2022-03-08T04:20:26.663 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2022-03-08 04:20:23.985649081 +0000 2022-03-08T04:20:26.663 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2022-03-08T04:20:26.671 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2022-03-08T04:20:26.672 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2022-03-08T04:20:26.672 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000604828 s, 847 kB/s 2022-03-08T04:20:26.675 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-08T04:20:26.682 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-08T04:20:26.683 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:26.683 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-08T04:20:26.683 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:26.683 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2022-03-08 04:20:24.053649079 +0000 2022-03-08T04:20:26.684 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2022-03-08 04:20:24.053649079 +0000 2022-03-08T04:20:26.684 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2022-03-08 04:20:24.053649079 +0000 2022-03-08T04:20:26.684 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2022-03-08T04:20:26.690 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2022-03-08T04:20:26.690 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2022-03-08T04:20:26.691 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000519241 s, 986 kB/s 2022-03-08T04:20:26.695 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-08T04:20:26.750 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-08T04:20:26.750 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:26.751 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-03-08T04:20:26.751 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:26.751 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2022-03-08 04:20:24.081649078 +0000 2022-03-08T04:20:26.751 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2022-03-08 04:20:24.081649078 +0000 2022-03-08T04:20:26.752 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2022-03-08 04:20:24.081649078 +0000 2022-03-08T04:20:26.752 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2022-03-08T04:20:26.760 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2022-03-08T04:20:26.760 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2022-03-08T04:20:26.760 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000478189 s, 1.1 MB/s 2022-03-08T04:20:26.764 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-08T04:20:26.772 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-08T04:20:26.772 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-08T04:20:26.772 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-08T04:20:26.773 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-08T04:20:26.773 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2022-03-08 04:20:24.113649078 +0000 2022-03-08T04:20:26.773 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2022-03-08 04:20:24.113649078 +0000 2022-03-08T04:20:26.773 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2022-03-08 04:20:24.113649078 +0000 2022-03-08T04:20:26.774 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2022-03-08T04:20:26.779 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2022-03-08T04:20:26.780 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2022-03-08T04:20:26.780 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000550897 s, 929 kB/s 2022-03-08T04:20:26.784 DEBUG:teuthology.orchestra.run.smithi105:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-08T04:20:26.874 INFO:teuthology.orchestra.run.smithi105.stdout:loop 2022-03-08T04:20:26.876 INFO:tasks.nvme_loop:Connecting nvme_loop smithi105:/dev/vg_nvme/lv_1... 2022-03-08T04:20:26.876 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-08T04:20:26.900 INFO:teuthology.orchestra.run.smithi105.stdout:1 2022-03-08T04:20:26.931 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/vg_nvme/lv_11 2022-03-08T04:20:26.948 INFO:tasks.nvme_loop:Connecting nvme_loop smithi105:/dev/vg_nvme/lv_2... 2022-03-08T04:20:26.949 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-08T04:20:26.970 INFO:teuthology.orchestra.run.smithi105.stdout:1 2022-03-08T04:20:26.995 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/vg_nvme/lv_21 2022-03-08T04:20:27.019 INFO:tasks.nvme_loop:Connecting nvme_loop smithi105:/dev/vg_nvme/lv_3... 2022-03-08T04:20:27.020 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-08T04:20:27.040 INFO:teuthology.orchestra.run.smithi105.stdout:1 2022-03-08T04:20:27.063 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/vg_nvme/lv_31 2022-03-08T04:20:27.087 INFO:tasks.nvme_loop:Connecting nvme_loop smithi105:/dev/vg_nvme/lv_4... 2022-03-08T04:20:27.089 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-08T04:20:27.157 INFO:teuthology.orchestra.run.smithi105.stdout:1 2022-03-08T04:20:27.178 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/vg_nvme/lv_41 2022-03-08T04:20:27.203 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:20:27.203 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:20:27.209 DEBUG:teuthology.orchestra.run.smithi105:> sudo nvme list 2022-03-08T04:20:27.220 INFO:teuthology.orchestra.run.smithi105.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T04:20:27.220 INFO:teuthology.orchestra.run.smithi105.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T04:20:27.221 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme0n1 PHFT620400TR400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T04:20:27.221 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme1n1 b0165f03b7837262 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:27.221 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme2n1 5ff6440f9cc7f155 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:27.221 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme3n1 7093b74e16b6fd1f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:27.223 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-08T04:20:28.224 DEBUG:teuthology.orchestra.run.smithi105:> sudo nvme list 2022-03-08T04:20:28.242 INFO:teuthology.orchestra.run.smithi105.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-08T04:20:28.243 INFO:teuthology.orchestra.run.smithi105.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-08T04:20:28.243 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme0n1 PHFT620400TR400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-08T04:20:28.243 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme1n1 b0165f03b7837262 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:28.243 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme2n1 5ff6440f9cc7f155 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:28.244 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme3n1 7093b74e16b6fd1f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:28.244 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme4n1 44f7cf85fbdf2b99 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-08T04:20:28.245 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-08T04:20:28.245 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:20:28.246 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/scratch_devs 2022-03-08T04:20:28.301 INFO:teuthology.run_tasks:Running task rook... 2022-03-08T04:20:28.396 INFO:tasks.rook:Rook start 2022-03-08T04:20:28.396 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3', 'rook_image': 'rook/ceph:master'} 2022-03-08T04:20:28.397 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-08T04:20:28.397 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-08T04:20:28.398 DEBUG:teuthology.orchestra.run.smithi018:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-08T04:20:28.406 INFO:teuthology.orchestra.run.smithi018.stderr:Cloning into 'rook'... 2022-03-08T04:20:31.010 INFO:tasks.rook:/home/teuthworker 2022-03-08T04:20:31.011 DEBUG:teuthology.orchestra.run.smithi018:> PATH=/sbin:/usr/sbin ip addr show 2022-03-08T04:20:31.016 INFO:teuthology.orchestra.run.smithi018.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-08T04:20:31.017 INFO:teuthology.orchestra.run.smithi018.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-08T04:20:31.017 INFO:teuthology.orchestra.run.smithi018.stdout: inet 127.0.0.1/8 scope host lo 2022-03-08T04:20:31.017 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft forever preferred_lft forever 2022-03-08T04:20:31.018 INFO:teuthology.orchestra.run.smithi018.stdout: inet6 ::1/128 scope host 2022-03-08T04:20:31.018 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft forever preferred_lft forever 2022-03-08T04:20:31.018 INFO:teuthology.orchestra.run.smithi018.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T04:20:31.019 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 0c:c4:7a:6b:fd:98 brd ff:ff:ff:ff:ff:ff 2022-03-08T04:20:31.019 INFO:teuthology.orchestra.run.smithi018.stdout: altname enp7s0f0 2022-03-08T04:20:31.019 INFO:teuthology.orchestra.run.smithi018.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T04:20:31.020 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 0c:c4:7a:6b:fd:99 brd ff:ff:ff:ff:ff:ff 2022-03-08T04:20:31.020 INFO:teuthology.orchestra.run.smithi018.stdout: altname enp7s0f1 2022-03-08T04:20:31.020 INFO:teuthology.orchestra.run.smithi018.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-08T04:20:31.020 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 0c:c4:7a:bd:13:62 brd ff:ff:ff:ff:ff:ff 2022-03-08T04:20:31.020 INFO:teuthology.orchestra.run.smithi018.stdout: altname enp1s0f0 2022-03-08T04:20:31.021 INFO:teuthology.orchestra.run.smithi018.stdout: inet 172.21.15.18/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-08T04:20:31.021 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft 42858sec preferred_lft 42858sec 2022-03-08T04:20:31.021 INFO:teuthology.orchestra.run.smithi018.stdout: inet6 fe80::ec4:7aff:febd:1362/64 scope link 2022-03-08T04:20:31.022 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft forever preferred_lft forever 2022-03-08T04:20:31.022 INFO:teuthology.orchestra.run.smithi018.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-08T04:20:31.022 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 0c:c4:7a:bd:13:63 brd ff:ff:ff:ff:ff:ff 2022-03-08T04:20:31.023 INFO:teuthology.orchestra.run.smithi018.stdout: altname enp1s0f1 2022-03-08T04:20:31.023 INFO:teuthology.orchestra.run.smithi018.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-08T04:20:31.023 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 02:42:1d:29:51:da brd ff:ff:ff:ff:ff:ff 2022-03-08T04:20:31.024 INFO:teuthology.orchestra.run.smithi018.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-08T04:20:31.025 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft forever preferred_lft forever 2022-03-08T04:20:31.026 DEBUG:teuthology.orchestra.run.smithi018:> which systemctl 2022-03-08T04:20:31.033 INFO:teuthology.orchestra.run.smithi018.stdout:/usr/bin/systemctl 2022-03-08T04:20:31.034 DEBUG:teuthology.orchestra.run.smithi018:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-08T04:20:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-08T04:20:31.083 DEBUG:teuthology.orchestra.run.smithi018:> true 2022-03-08T04:20:31.117 DEBUG:teuthology.orchestra.run.smithi018:> sudo lsb_release -is 2022-03-08T04:20:31.169 INFO:teuthology.orchestra.run.smithi018.stdout:Ubuntu 2022-03-08T04:20:31.178 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-08T04:20:31.179 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-08T04:20:31.191 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T04:20:31.206 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:20:31.207 DEBUG:teuthology.orchestra.run.smithi018:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-08T04:20:31.216 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-08T04:20:31.258 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:20:31.259 DEBUG:teuthology.orchestra.run.smithi018:> dd of=operator.yaml 2022-03-08T04:20:31.267 INFO:tasks.rook:Deploying operator 2022-03-08T04:20:31.279 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T04:20:31.348 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T04:20:31.359 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-08T04:20:31.662 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-08T04:20:31.684 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-08T04:20:31.711 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-08T04:20:31.760 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-08T04:20:31.876 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-08T04:20:31.965 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-08T04:20:32.024 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-08T04:20:32.043 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-08T04:20:32.139 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-08T04:20:32.143 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-08T04:20:32.235 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-08T04:20:32.303 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-08T04:20:32.319 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-08T04:20:32.335 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-08T04:20:32.355 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-08T04:20:32.378 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-08T04:20:32.481 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-08T04:20:32.528 INFO:teuthology.orchestra.run.smithi018.stdout:namespace/rook-ceph created 2022-03-08T04:20:32.553 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T04:20:32.570 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-08T04:20:32.629 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-08T04:20:32.653 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T04:20:32.678 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-08T04:20:32.695 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T04:20:32.715 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T04:20:32.804 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T04:20:32.807 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T04:20:32.837 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T04:20:32.864 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T04:20:32.867 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T04:20:32.887 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-08T04:20:32.892 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-08T04:20:32.970 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T04:20:32.975 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-08T04:20:32.983 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-08T04:20:32.991 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-08T04:20:33.012 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-08T04:20:33.016 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T04:20:33.025 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T04:20:33.033 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-08T04:20:33.041 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-08T04:20:33.049 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-08T04:20:33.111 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-08T04:20:33.114 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-08T04:20:33.125 INFO:teuthology.orchestra.run.smithi018.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-08T04:20:33.125 INFO:teuthology.orchestra.run.smithi018.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-08T04:20:33.133 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-08T04:20:33.175 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-08T04:20:33.183 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-08T04:20:33.204 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T04:20:33.208 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T04:20:33.261 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T04:20:33.264 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T04:20:33.275 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T04:20:33.296 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-08T04:20:33.308 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-08T04:20:33.317 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-08T04:20:33.324 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-08T04:20:33.333 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-08T04:20:33.341 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-08T04:20:33.404 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-08T04:20:33.407 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-08T04:20:33.416 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-08T04:20:33.424 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-08T04:20:33.433 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-08T04:20:33.441 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-08T04:20:33.450 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-08T04:20:33.545 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-08T04:20:33.549 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-08T04:20:33.558 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-mgr created 2022-03-08T04:20:33.567 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-osd created 2022-03-08T04:20:33.590 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-08T04:20:33.614 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-system created 2022-03-08T04:20:33.630 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-08T04:20:33.697 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-08T04:20:33.708 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-08T04:20:33.730 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-08T04:20:33.756 INFO:teuthology.orchestra.run.smithi018.stdout:configmap/rook-ceph-operator-config created 2022-03-08T04:20:33.854 INFO:teuthology.orchestra.run.smithi018.stdout:deployment.apps/rook-ceph-operator created 2022-03-08T04:20:33.858 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T04:20:33.944 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:20:33.944 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-hqvwz 0/1 Pending 0 0s 2022-03-08T04:20:43.947 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T04:20:44.022 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:20:44.025 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-hqvwz 0/1 Pending 0 11s 2022-03-08T04:20:54.027 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T04:20:54.097 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:20:54.098 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-hqvwz 0/1 Pending 0 21s 2022-03-08T04:21:04.102 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T04:21:04.167 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:21:04.168 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-hqvwz 0/1 ContainerCreating 0 31s 2022-03-08T04:21:14.171 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T04:21:14.237 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:21:14.238 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-hqvwz 0/1 ContainerCreating 0 41s 2022-03-08T04:21:24.241 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T04:21:24.307 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:21:24.308 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-hqvwz 0/1 ContainerCreating 0 51s 2022-03-08T04:21:34.310 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T04:21:34.376 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:21:34.376 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-hqvwz 0/1 ContainerCreating 0 61s 2022-03-08T04:21:44.379 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-08T04:21:44.463 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:21:44.464 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-hqvwz 1/1 Running 0 71s 2022-03-08T04:21:44.465 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-hqvwz 2022-03-08T04:21:44.480 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:21:44.481 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:21:44.488 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:21:44.489 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:21:44.497 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:21:44.498 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2022-03-08T04:21:44.507 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-08T04:21:44.508 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-08T04:21:44.508 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-08T04:21:44.509 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-08T04:21:44.509 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-08T04:21:44.509 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-08T04:21:44.510 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-08T04:21:44.510 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-08T04:21:44.513 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f - 2022-03-08T04:21:44.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:42.875885 I | rookcmd: starting Rook v1.8.0-alpha.0.381.g235781e28 with arguments '/usr/local/bin/rook ceph operator' 2022-03-08T04:21:44.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:42.876006 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-08T04:21:44.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:42.876009 I | cephcmd: starting Rook-Ceph operator 2022-03-08T04:21:44.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.017723 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-08T04:21:44.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.026323 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T04:21:44.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.026343 I | operator: watching all namespaces for Ceph CRs 2022-03-08T04:21:44.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.026419 I | operator: setting up schemes 2022-03-08T04:21:44.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.028061 I | operator: setting up the controller-runtime manager 2022-03-08T04:21:44.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.334180 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T04:21:44.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337384 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T04:21:44.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337432 I | ceph-cluster-controller: successfully started 2022-03-08T04:21:44.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337516 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T04:21:44.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337554 I | ceph-crashcollector-controller: successfully started 2022-03-08T04:21:44.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337587 I | ceph-block-pool-controller: successfully started 2022-03-08T04:21:44.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337632 I | ceph-object-store-user-controller: successfully started 2022-03-08T04:21:44.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337667 I | ceph-object-realm-controller: successfully started 2022-03-08T04:21:44.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337690 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T04:21:44.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337717 I | ceph-object-zone-controller: successfully started 2022-03-08T04:21:44.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337854 I | ceph-object-controller: successfully started 2022-03-08T04:21:44.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337903 I | ceph-file-controller: successfully started 2022-03-08T04:21:44.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.337962 I | ceph-nfs-controller: successfully started 2022-03-08T04:21:44.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338004 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T04:21:44.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338043 I | ceph-client-controller: successfully started 2022-03-08T04:21:44.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338067 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T04:21:44.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338104 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T04:21:44.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338141 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T04:21:44.566 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338163 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T04:21:44.566 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338182 I | ceph-bucket-topic: successfully started 2022-03-08T04:21:44.566 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338207 I | ceph-bucket-notification: successfully started 2022-03-08T04:21:44.566 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338225 I | ceph-bucket-notification: successfully started 2022-03-08T04:21:44.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.338239 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T04:21:44.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.339249 I | operator: starting the controller-runtime manager 2022-03-08T04:21:44.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.442459 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T04:21:44.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.442485 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T04:21:44.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.442494 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T04:21:44.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.445222 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-08T04:21:44.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.445240 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-08T04:21:44.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.461911 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-08T04:21:44.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.461960 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T04:21:44.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.497467 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-08T04:21:44.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.497493 I | ceph-csi: CSI CephFS driver disabled 2022-03-08T04:21:44.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.497503 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-08T04:21:44.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.506378 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-08T04:21:44.570 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:43.516468 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-08T04:21:45.591 INFO:teuthology.orchestra.run.smithi018.stdout:configmap/rook-config-override created 2022-03-08T04:21:45.595 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-08T04:21:45.595 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:21:45.596 DEBUG:teuthology.orchestra.run.smithi018:> dd of=cluster.yaml 2022-03-08T04:21:45.603 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f cluster.yaml 2022-03-08T04:21:48.008 INFO:teuthology.orchestra.run.smithi018.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-08T04:21:48.013 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.015074 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-08T04:21:48.021 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T04:21:48.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.031438 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T04:21:48.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.038861 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-08T04:21:48.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.038900 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T04:21:48.042 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-08T04:21:48.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096783 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T04:21:48.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096811 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T04:21:48.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096823 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T04:21:48.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096834 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T04:21:48.097 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096846 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T04:21:48.097 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096860 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T04:21:48.097 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096875 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T04:21:48.098 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096890 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T04:21:48.098 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096906 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T04:21:48.098 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096922 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T04:21:48.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096934 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T04:21:48.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096948 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T04:21:48.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096963 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T04:21:48.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096975 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T04:21:48.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.096987 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T04:21:48.103 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.097001 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T04:21:48.103 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.097130 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:21:48.103 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.097147 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:21:48.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:21:48.099181 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T04:21:48.402 INFO:teuthology.orchestra.run.smithi018.stdout:deployment.apps/rook-ceph-tools created 2022-03-08T04:21:48.411 INFO:tasks.rook:Waiting for tools container to start 2022-03-08T04:21:48.412 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:21:48.495 INFO:teuthology.orchestra.run.smithi018.stderr:No resources found in rook-ceph namespace. 2022-03-08T04:21:53.501 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:21:53.561 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:21:53.561 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 5s 2022-03-08T04:21:58.564 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:21:58.642 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:21:58.642 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 10s 2022-03-08T04:22:03.649 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:03.723 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:03.723 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 15s 2022-03-08T04:22:08.725 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:08.807 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:08.808 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 20s 2022-03-08T04:22:13.811 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:13.877 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:13.878 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 25s 2022-03-08T04:22:18.881 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:18.953 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:18.953 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 30s 2022-03-08T04:22:23.955 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:24.022 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:24.023 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 36s 2022-03-08T04:22:29.024 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:29.086 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:29.087 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 41s 2022-03-08T04:22:34.090 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:34.159 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:34.159 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 46s 2022-03-08T04:22:39.161 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:39.240 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:39.241 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 51s 2022-03-08T04:22:44.243 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:44.306 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:44.307 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 56s 2022-03-08T04:22:49.310 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:49.377 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:49.378 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 61s 2022-03-08T04:22:54.382 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:54.464 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:54.464 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 66s 2022-03-08T04:22:59.467 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:22:59.536 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:22:59.537 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 71s 2022-03-08T04:23:04.540 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:04.615 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:04.615 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 76s 2022-03-08T04:23:09.619 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:09.703 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:09.704 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 81s 2022-03-08T04:23:14.708 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:14.783 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:14.784 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 86s 2022-03-08T04:23:19.787 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:19.854 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:19.854 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 91s 2022-03-08T04:23:23.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386821 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T04:23:23.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386860 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T04:23:23.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386872 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T04:23:23.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386880 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T04:23:23.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386887 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T04:23:23.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386900 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T04:23:23.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386906 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T04:23:23.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386913 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T04:23:23.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386920 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T04:23:23.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386929 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T04:23:23.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386941 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T04:23:23.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386952 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T04:23:23.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386959 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T04:23:23.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386966 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T04:23:23.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386972 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T04:23:23.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386980 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T04:23:23.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386987 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T04:23:23.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.386994 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T04:23:23.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.387001 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T04:23:23.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.387009 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T04:23:23.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.387021 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T04:23:23.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.387033 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T04:23:23.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.387040 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T04:23:23.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.387048 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T04:23:23.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.387055 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T04:23:23.421 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.422003 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T04:23:23.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.432354 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:23:23.432 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.432378 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:23:23.432 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.432384 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:23:23.433 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.432388 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:23:23.433 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.432393 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:23:23.433 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.432398 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:23:23.433 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.432402 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T04:23:23.527 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.528055 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:23:23.527 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.528089 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:23:23.527 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.528098 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T04:23:23.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.562896 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T04:23:23.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.684897 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:23:23.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.684945 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:23:23.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.684962 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T04:23:23.859 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.860419 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:23:23.861 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.860449 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:23:23.861 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.860462 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T04:23:23.906 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:23.906460 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T04:23:24.409 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:24.409893 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T04:23:24.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:24.766509 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T04:23:24.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:24.766539 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T04:23:24.857 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:25.258 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:25.258 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 97s 2022-03-08T04:23:25.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:25.259360 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T04:23:30.261 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:30.388 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:30.389 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 102s 2022-03-08T04:23:35.391 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:35.460 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:35.460 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 107s 2022-03-08T04:23:40.465 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:40.550 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:40.550 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 112s 2022-03-08T04:23:45.553 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:45.624 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:45.624 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 117s 2022-03-08T04:23:50.628 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:50.697 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:50.697 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m2s 2022-03-08T04:23:55.700 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:23:55.775 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:23:55.775 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m7s 2022-03-08T04:23:56.594 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:56.595370 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T04:23:56.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:56.595399 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T04:23:56.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:56.595412 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T04:23:56.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:56.688373 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T04:23:56.720 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:56.721909 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-08 04:23:56.699528742 +0000 UTC m=+133.843595849 LastTransitionTime:2022-03-08 04:23:56.699528499 +0000 UTC m=+133.843595655}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:23:56.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:56.778245 I | op-mon: start running mons 2022-03-08T04:23:56.930 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:56.931751 I | op-mon: creating mon secrets for a new cluster 2022-03-08T04:23:56.998 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:56.999807 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-08T04:23:57.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:57.002555 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-08T04:23:57.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:57.550682 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:23:57.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:57.550900 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:23:58.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:58.756774 I | op-mon: parsing mon endpoints: 2022-03-08T04:23:58.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:58.756808 W | op-mon: ignoring invalid monitor 2022-03-08T04:23:58.756 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:58.756844 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T04:23:58.756 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:58.756860 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T04:23:58.759 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:58.759009 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T04:23:58.760 INFO:tasks.rook.operator.smithi018.stdout:I0308 04:23:58.759141 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T04:23:59.550 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:59.552276 I | op-mon: targeting the mon count 3 2022-03-08T04:23:59.558 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:59.559994 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-08T04:23:59.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:59.590908 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-08T04:23:59.614 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:23:59.616243 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-08T04:24:00.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:00.350242 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi105 2022-03-08T04:24:00.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:00.350268 I | op-mon: mon a assigned to node smithi105 2022-03-08T04:24:00.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:00.551567 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi084 2022-03-08T04:24:00.550 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:00.551591 I | op-mon: mon b assigned to node smithi084 2022-03-08T04:24:00.778 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:01.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:01.180752 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi018 2022-03-08T04:24:01.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:01.180792 I | op-mon: mon c assigned to node smithi018 2022-03-08T04:24:01.185 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:01.185 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m13s 2022-03-08T04:24:01.190 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:01.191659 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-08T04:24:01.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:01.489070 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-08T04:24:02.049 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:02.051219 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-08T04:24:02.540 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:02.541739 I | op-mon: creating mon a 2022-03-08T04:24:03.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:03.661871 I | op-mon: mon "a" endpoint is [v2:10.104.177.194:3300,v1:10.104.177.194:6789] 2022-03-08T04:24:04.513 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:04.515162 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T04:24:04.519 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:04.515382 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-08T04:24:04.519 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:04.515742 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.104.177.194:6789"]}] data:a=10.104.177.194:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:-1] 2022-03-08T04:24:05.222 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:05.223467 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:24:05.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:05.223750 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:24:06.187 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:06.337 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:06.337 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m18s 2022-03-08T04:24:06.518 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:06.520263 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-08T04:24:07.058 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:07.060393 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-08T04:24:07.443 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:07.445145 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.104.177.194:6789"]}] data:a=10.104.177.194:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:0] 2022-03-08T04:24:07.444 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:07.445188 I | op-mon: waiting for mon quorum with [a] 2022-03-08T04:24:07.521 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:07.522935 I | op-mon: mon a is not yet running 2022-03-08T04:24:07.522 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:07.522965 I | op-mon: mons running: [] 2022-03-08T04:24:11.339 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:11.446 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:11.447 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m23s 2022-03-08T04:24:16.449 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:16.523 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:16.524 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m28s 2022-03-08T04:24:21.526 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:21.596 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:21.596 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m33s 2022-03-08T04:24:26.599 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:26.661 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:26.661 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m38s 2022-03-08T04:24:27.670 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:27.672862 I | op-mon: mons running: [a] 2022-03-08T04:24:31.665 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:31.780 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:31.781 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m43s 2022-03-08T04:24:36.783 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:37.040 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:37.040 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m49s 2022-03-08T04:24:39.554 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:39.556951 I | op-mon: Monitors in quorum: [a] 2022-03-08T04:24:39.555 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:39.556981 I | op-mon: mons created: 1 2022-03-08T04:24:39.791 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:39.793648 I | op-mon: waiting for mon quorum with [a] 2022-03-08T04:24:39.982 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:39.985313 I | op-mon: mons running: [a] 2022-03-08T04:24:40.273 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:40.276048 I | op-mon: Monitors in quorum: [a] 2022-03-08T04:24:40.274 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:40.276082 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:24:40.617 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:40.619687 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:24:40.617 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:40.619723 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:24:40.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:40.904653 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:24:40.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:40.904722 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:24:41.176 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:41.178928 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:24:41.177 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:41.178963 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:24:41.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:41.883454 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:24:41.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:41.883486 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T04:24:42.043 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:42.110 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:42.112913 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T04:24:42.110 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:42.112933 I | op-mon: creating mon b 2022-03-08T04:24:42.196 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:42.197 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 2m54s 2022-03-08T04:24:43.570 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:43.573309 I | op-mon: mon "a" endpoint is [v2:10.104.177.194:3300,v1:10.104.177.194:6789] 2022-03-08T04:24:44.177 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:44.179648 I | op-mon: mon "b" endpoint is [v2:10.99.92.113:3300,v1:10.99.92.113:6789] 2022-03-08T04:24:45.165 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:45.167975 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.104.177.194:6789","10.99.92.113:6789"]}] data:b=10.99.92.113:6789,a=10.104.177.194:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:0] 2022-03-08T04:24:46.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:46.459096 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:24:46.460 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:46.459610 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:24:47.201 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:48.031 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:48.031 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m 2022-03-08T04:24:48.624 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:48.626978 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T04:24:49.044 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:49.046858 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T04:24:49.045 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:49.046896 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T04:24:49.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:49.402001 I | op-mon: mon b is not yet running 2022-03-08T04:24:49.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:49.402041 I | op-mon: mons running: [a] 2022-03-08T04:24:49.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:49.625184 I | op-mon: Monitors in quorum: [a] 2022-03-08T04:24:49.836 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:49.833460 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-08T04:24:51.828 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:51.829438 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.104.177.194:6789","10.99.92.113:6789"]}] data:a=10.104.177.194:6789,b=10.99.92.113:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:1] 2022-03-08T04:24:51.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:51.829491 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T04:24:52.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:52.970311 I | op-mon: mon b is not yet running 2022-03-08T04:24:52.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:52.970349 I | op-mon: mons running: [a] 2022-03-08T04:24:53.038 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:53.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:53.190731 I | op-mon: Monitors in quorum: [a] 2022-03-08T04:24:53.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:53.190771 I | op-mon: mons created: 2 2022-03-08T04:24:53.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:53.409082 I | op-mon: waiting for mon quorum with [a b] 2022-03-08T04:24:53.773 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:53.773 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m5s 2022-03-08T04:24:53.860 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:53.863418 I | op-mon: mon b is not yet running 2022-03-08T04:24:53.861 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:53.863445 I | op-mon: mons running: [a] 2022-03-08T04:24:58.776 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:24:58.874 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:24:58.874 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m10s 2022-03-08T04:24:59.455 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:59.458821 I | op-mon: mon b is not yet running 2022-03-08T04:24:59.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:24:59.458851 I | op-mon: mons running: [a] 2022-03-08T04:25:03.876 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:03.945 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:03.946 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m15s 2022-03-08T04:25:04.668 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:04.671121 I | op-mon: mon b is not yet running 2022-03-08T04:25:04.668 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:04.671147 I | op-mon: mons running: [a] 2022-03-08T04:25:08.948 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:09.056 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:09.056 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m21s 2022-03-08T04:25:09.708 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:09.710537 I | op-mon: mon b is not yet running 2022-03-08T04:25:09.708 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:09.710563 I | op-mon: mons running: [a] 2022-03-08T04:25:14.058 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:14.128 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:14.129 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m26s 2022-03-08T04:25:14.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:14.779461 I | op-mon: mon b is not yet running 2022-03-08T04:25:14.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:14.779494 I | op-mon: mons running: [a] 2022-03-08T04:25:19.131 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:19.199 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:19.199 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m31s 2022-03-08T04:25:19.806 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:19.809786 I | op-mon: mon b is not yet running 2022-03-08T04:25:19.806 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:19.809817 I | op-mon: mons running: [a] 2022-03-08T04:25:24.202 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:24.265 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:24.265 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m36s 2022-03-08T04:25:24.841 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:24.844113 I | op-mon: mon b is not yet running 2022-03-08T04:25:24.841 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:24.844137 I | op-mon: mons running: [a] 2022-03-08T04:25:29.269 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:29.343 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:29.344 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m41s 2022-03-08T04:25:29.860 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:29.862225 I | op-mon: mons running: [a b] 2022-03-08T04:25:34.347 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:34.416 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:34.416 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m46s 2022-03-08T04:25:36.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:36.110362 I | op-mon: Monitors in quorum: [a b] 2022-03-08T04:25:36.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:36.110400 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:25:36.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:36.339404 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:25:36.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:36.339448 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:25:36.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:36.572814 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:25:36.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:36.572845 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:25:36.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:36.799965 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:25:36.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:36.799993 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:25:37.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:37.010698 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:25:37.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:37.010730 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T04:25:37.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:37.229883 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T04:25:37.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:37.229911 I | op-mon: creating mon c 2022-03-08T04:25:37.423 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:37.427444 I | op-mon: mon "a" endpoint is [v2:10.104.177.194:3300,v1:10.104.177.194:6789] 2022-03-08T04:25:38.116 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:38.119956 I | op-mon: mon "b" endpoint is [v2:10.99.92.113:3300,v1:10.99.92.113:6789] 2022-03-08T04:25:38.213 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:38.216642 I | op-mon: mon "c" endpoint is [v2:10.110.14.128:3300,v1:10.110.14.128:6789] 2022-03-08T04:25:38.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:38.364103 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.104.177.194:6789","10.99.92.113:6789","10.110.14.128:6789"]}] data:a=10.104.177.194:6789,b=10.99.92.113:6789,c=10.110.14.128:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:1] 2022-03-08T04:25:38.396 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:38.399727 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:25:38.396 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:38.399944 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:25:38.658 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:38.662388 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T04:25:38.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:38.672836 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T04:25:38.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:38.672875 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T04:25:39.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:39.239288 I | op-mon: mons running: [a b c] 2022-03-08T04:25:39.419 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:39.485 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:39.485 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m51s 2022-03-08T04:25:39.486 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:39.489561 I | op-mon: Monitors in quorum: [a b] 2022-03-08T04:25:39.491 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:39.495931 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T04:25:39.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:39.538975 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T04:25:39.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:39.539004 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T04:25:39.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:39.836692 I | op-mon: mon c is not yet running 2022-03-08T04:25:39.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:39.836722 I | op-mon: mons running: [a b] 2022-03-08T04:25:40.075 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:40.078541 I | op-mon: Monitors in quorum: [a b] 2022-03-08T04:25:40.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:40.126345 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-08T04:25:40.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:40.951011 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.104.177.194:6789","10.99.92.113:6789","10.110.14.128:6789"]}] data:a=10.104.177.194:6789,b=10.99.92.113:6789,c=10.110.14.128:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:2] 2022-03-08T04:25:40.948 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:40.951048 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T04:25:41.436 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:41.439683 I | op-mon: mon c is not yet running 2022-03-08T04:25:41.436 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:41.439724 I | op-mon: mons running: [a b] 2022-03-08T04:25:41.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:41.667208 I | op-mon: Monitors in quorum: [a b] 2022-03-08T04:25:41.664 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:41.667232 I | op-mon: mons created: 3 2022-03-08T04:25:41.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:41.882975 I | op-mon: waiting for mon quorum with [a b c] 2022-03-08T04:25:42.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:42.038014 I | op-mon: mon c is not yet running 2022-03-08T04:25:42.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:42.038050 I | op-mon: mons running: [a b] 2022-03-08T04:25:44.488 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:44.570 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:44.571 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 3m56s 2022-03-08T04:25:47.052 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:47.056143 I | op-mon: mon c is not yet running 2022-03-08T04:25:47.052 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:47.056174 I | op-mon: mons running: [a b] 2022-03-08T04:25:49.573 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:49.646 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:49.647 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 4m1s 2022-03-08T04:25:52.073 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:52.077259 I | op-mon: mon c is not yet running 2022-03-08T04:25:52.073 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:52.077291 I | op-mon: mons running: [a b] 2022-03-08T04:25:54.649 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:54.718 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:54.718 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 4m6s 2022-03-08T04:25:57.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:57.135710 I | op-mon: mon c is not yet running 2022-03-08T04:25:57.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:25:57.135743 I | op-mon: mons running: [a b] 2022-03-08T04:25:59.721 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:25:59.787 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:25:59.788 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 0/1 ContainerCreating 0 4m11s 2022-03-08T04:26:02.691 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:02.694943 I | op-mon: mon c is not yet running 2022-03-08T04:26:02.691 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:02.694979 I | op-mon: mons running: [a b] 2022-03-08T04:26:04.790 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-08T04:26:05.113 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-08T04:26:05.113 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-n929f 1/1 Running 0 4m17s 2022-03-08T04:26:05.115 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-08T04:26:05.116 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch status -f json 2022-03-08T04:26:05.623 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:26:05.624 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-08T04:26:05.624 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-08T04:26:05.624 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-08T04:26:05.624 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T04:26:05.625 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T04:26:05.625 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-08T04:26:05.625 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-08T04:26:05.625 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-08T04:26:05.626 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-08T04:26:05.626 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-08T04:26:05.626 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T04:26:05.627 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-08T04:26:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T04:26:05.630 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-08T04:26:07.721 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:07.722951 I | op-mon: mon c is not yet running 2022-03-08T04:26:07.722 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:07.722985 I | op-mon: mons running: [a b] 2022-03-08T04:26:13.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:13.553585 I | op-mon: mon c is not yet running 2022-03-08T04:26:13.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:13.553613 I | op-mon: mons running: [a b] 2022-03-08T04:26:15.631 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch status -f json 2022-03-08T04:26:16.152 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:26:16.152 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-08T04:26:16.152 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-08T04:26:16.153 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-08T04:26:16.153 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T04:26:16.153 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T04:26:16.153 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-08T04:26:16.154 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-08T04:26:16.154 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-08T04:26:16.154 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-08T04:26:16.154 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-08T04:26:16.155 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T04:26:16.155 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-08T04:26:16.156 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-08T04:26:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T04:26:18.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:18.574050 I | op-mon: mon c is not yet running 2022-03-08T04:26:18.570 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:18.574085 I | op-mon: mons running: [a b] 2022-03-08T04:26:23.596 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:23.601227 I | op-mon: mons running: [a b c] 2022-03-08T04:26:26.161 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch status -f json 2022-03-08T04:26:26.683 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:26:26.683 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-08T04:26:26.684 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-08T04:26:26.684 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-08T04:26:26.684 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T04:26:26.684 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T04:26:26.685 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-08T04:26:26.685 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-08T04:26:26.685 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-08T04:26:26.685 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-08T04:26:26.685 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-08T04:26:26.686 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T04:26:26.686 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-08T04:26:26.687 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-08T04:26:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T04:26:26.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:26.837634 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:26:26.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:26.837676 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:26:27.054 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:27.058873 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:26:27.055 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:27.058905 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:26:27.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:27.286438 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:26:27.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:27.286463 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:26:27.507 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:27.511633 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:26:27.507 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:27.511668 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:26:27.808 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:27.812362 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:26:27.808 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:27.812402 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T04:26:28.030 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:28.034743 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T04:26:28.031 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:28.034795 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T04:26:28.486 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:28.490848 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T04:26:28.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:28.838761 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T04:26:29.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:29.232334 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T04:26:29.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:29.686819 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T04:26:29.683 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:29.686851 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T04:26:30.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:30.248187 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T04:26:30.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:30.475498 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T04:26:30.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:30.475535 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T04:26:30.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:30.475547 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T04:26:30.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:30.875176 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T04:26:30.959 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:30.963841 I | op-mgr: start running mgr 2022-03-08T04:26:30.959 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:30.963879 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T04:26:36.691 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch status -f json 2022-03-08T04:26:37.256 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:26:37.256 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-08T04:26:37.257 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-08T04:26:37.257 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-08T04:26:37.257 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T04:26:37.257 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T04:26:37.258 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-08T04:26:37.258 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-08T04:26:37.258 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-08T04:26:37.258 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-08T04:26:37.259 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-08T04:26:37.259 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T04:26:37.259 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-08T04:26:37.260 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-08T04:26:37.262 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T04:26:47.264 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch status -f json 2022-03-08T04:26:47.750 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:26:47.750 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-08T04:26:47.751 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-08T04:26:47.751 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-08T04:26:47.751 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-08T04:26:47.751 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-08T04:26:47.752 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-08T04:26:47.752 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-08T04:26:47.752 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-08T04:26:47.752 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-08T04:26:47.753 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-08T04:26:47.753 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-08T04:26:47.753 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-08T04:26:47.755 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-08T04:26:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-08T04:26:52.999 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.004279 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-08T04:26:53.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.008260 I | op-mgr: setting services to point to mgr "a" 2022-03-08T04:26:53.078 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.082149 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T04:26:53.079 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.083064 I | op-mgr: successful modules: balancer 2022-03-08T04:26:53.148 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.153675 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T04:26:53.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.153706 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-08T04:26:53.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.153717 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T04:26:53.152 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.158221 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T04:26:53.158 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.161317 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-08T04:26:53.158 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.161343 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T04:26:53.158 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.161348 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T04:26:53.705 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.710053 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T04:26:53.705 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.710076 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T04:26:53.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.721374 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-08T04:26:53.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.721411 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-08T04:26:53.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:53.721424 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-08T04:26:54.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:54.248301 I | op-mgr: successful modules: dashboard 2022-03-08T04:26:55.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:55.166363 I | op-mgr: successful modules: prometheus 2022-03-08T04:26:55.167 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:26:55.172775 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T04:26:57.760 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch status -f json 2022-03-08T04:27:04.373 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:04.378534 I | op-mgr: successful modules: orchestrator modules 2022-03-08T04:27:04.424 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:27:04.425 INFO:teuthology.orchestra.run.smithi018.stdout:{"available": true, "backend": "rook"} 2022-03-08T04:27:04.449 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-08T04:27:04.449 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-08T04:27:05.131 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch apply osd --all-available-devices 2022-03-08T04:27:05.937 INFO:teuthology.orchestra.run.smithi018.stdout:['Success'] 2022-03-08T04:27:05.939 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.942944 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T04:27:05.939 INFO:tasks.rook.operator.smithi018.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T04:27:05.939 INFO:tasks.rook.operator.smithi018.stdout: Storage: v1.StorageScopeSpec{ 2022-03-08T04:27:05.940 INFO:tasks.rook.operator.smithi018.stdout: ... // 3 identical fields 2022-03-08T04:27:05.940 INFO:tasks.rook.operator.smithi018.stdout: Config: nil, 2022-03-08T04:27:05.940 INFO:tasks.rook.operator.smithi018.stdout: Selection: {}, 2022-03-08T04:27:05.940 INFO:tasks.rook.operator.smithi018.stdout:- StorageClassDeviceSets: nil, 2022-03-08T04:27:05.941 INFO:tasks.rook.operator.smithi018.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T04:27:05.941 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.941 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi018-lv-1", 2022-03-08T04:27:05.948 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.949 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.949 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.949 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.950 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi018-lv-2", 2022-03-08T04:27:05.950 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.950 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.950 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.951 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.951 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi018-lv-3", 2022-03-08T04:27:05.951 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.951 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.951 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.952 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.952 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi018-lv-4", 2022-03-08T04:27:05.952 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.952 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.953 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.953 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.953 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi084-lv-1", 2022-03-08T04:27:05.953 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.954 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.954 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.954 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.954 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi084-lv-2", 2022-03-08T04:27:05.955 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.955 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.955 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.955 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.956 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi084-lv-3", 2022-03-08T04:27:05.956 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.956 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.956 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.957 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.957 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi084-lv-4", 2022-03-08T04:27:05.957 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.957 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.958 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.958 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.958 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi105-lv-1", 2022-03-08T04:27:05.958 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.959 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.959 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.959 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.959 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi105-lv-2", 2022-03-08T04:27:05.960 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.960 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.960 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.960 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.960 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi105-lv-3", 2022-03-08T04:27:05.961 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.961 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.961 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.961 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:27:05.962 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi105-lv-4", 2022-03-08T04:27:05.962 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:27:05.963 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-08T04:27:05.963 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.963 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:27:05.963 INFO:tasks.rook.operator.smithi018.stdout: }, 2022-03-08T04:27:05.964 INFO:tasks.rook.operator.smithi018.stdout: Annotations: nil, 2022-03-08T04:27:05.964 INFO:tasks.rook.operator.smithi018.stdout: Labels: nil, 2022-03-08T04:27:05.964 INFO:tasks.rook.operator.smithi018.stdout: ... // 20 identical fields 2022-03-08T04:27:05.964 INFO:tasks.rook.operator.smithi018.stdout: } 2022-03-08T04:27:05.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.943076 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T04:27:05.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.943214 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-08T04:27:05.965 INFO:tasks.rook.operator.smithi018.stdout:I0308 04:27:05.943346 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T04:27:05.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.943559 I | operator: successfully started the controller-runtime manager 2022-03-08T04:27:05.966 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.943592 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-08T04:27:05.966 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.943666 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-08T04:27:05.967 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-08T04:27:05.968 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:27:05.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.984371 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T04:27:05.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.984400 I | operator: watching all namespaces for Ceph CRs 2022-03-08T04:27:05.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.984427 I | operator: setting up schemes 2022-03-08T04:27:05.982 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:05.987039 I | operator: setting up the controller-runtime manager 2022-03-08T04:27:06.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.291874 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T04:27:06.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.294679 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T04:27:06.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.294710 I | ceph-cluster-controller: successfully started 2022-03-08T04:27:06.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.294765 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T04:27:06.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.294789 I | ceph-crashcollector-controller: successfully started 2022-03-08T04:27:06.292 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.294818 I | ceph-block-pool-controller: successfully started 2022-03-08T04:27:06.292 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.294837 I | ceph-object-store-user-controller: successfully started 2022-03-08T04:27:06.292 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.294859 I | ceph-object-realm-controller: successfully started 2022-03-08T04:27:06.292 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.294870 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T04:27:06.293 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.294883 I | ceph-object-zone-controller: successfully started 2022-03-08T04:27:06.293 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295017 I | ceph-object-controller: successfully started 2022-03-08T04:27:06.294 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295056 I | ceph-file-controller: successfully started 2022-03-08T04:27:06.294 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295095 I | ceph-nfs-controller: successfully started 2022-03-08T04:27:06.294 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295126 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T04:27:06.294 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295153 I | ceph-client-controller: successfully started 2022-03-08T04:27:06.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295170 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T04:27:06.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295195 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T04:27:06.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295214 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T04:27:06.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295235 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T04:27:06.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295256 I | ceph-bucket-topic: successfully started 2022-03-08T04:27:06.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295272 I | ceph-bucket-notification: successfully started 2022-03-08T04:27:06.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295283 I | ceph-bucket-notification: successfully started 2022-03-08T04:27:06.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.295298 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T04:27:06.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.296394 I | operator: starting the controller-runtime manager 2022-03-08T04:27:06.399 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.404261 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T04:27:06.399 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.404327 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T04:27:06.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.404345 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T04:27:06.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.404413 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T04:27:06.403 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.409016 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T04:27:06.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.410949 I | op-mon: parsing mon endpoints: a=10.104.177.194:6789,b=10.99.92.113:6789,c=10.110.14.128:6789 2022-03-08T04:27:06.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.411453 I | op-mon: parsing mon endpoints: a=10.104.177.194:6789,b=10.99.92.113:6789,c=10.110.14.128:6789 2022-03-08T04:27:06.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.411489 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T04:27:06.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.411498 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T04:27:06.409 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.412510 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T04:27:06.409 INFO:tasks.rook.operator.smithi018.stdout:I0308 04:27:06.412563 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T04:27:06.422 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.427919 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T04:27:06.426 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.432496 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T04:27:06.498 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:27:06.516 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T04:27:06.517 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T04:27:06.554 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558408 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T04:27:06.555 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558444 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T04:27:06.555 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558453 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T04:27:06.555 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558461 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T04:27:06.555 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558471 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T04:27:06.556 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558483 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T04:27:06.556 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558497 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T04:27:06.556 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558510 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T04:27:06.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558518 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T04:27:06.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558528 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T04:27:06.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558542 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T04:27:06.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558550 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T04:27:06.558 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558557 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T04:27:06.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558563 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T04:27:06.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558569 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T04:27:06.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558583 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T04:27:06.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558664 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:27:06.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:06.558688 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:27:12.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148470 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T04:27:12.150 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148506 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T04:27:12.150 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148514 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T04:27:12.150 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148518 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T04:27:12.150 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148522 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T04:27:12.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148527 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T04:27:12.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148532 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T04:27:12.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148535 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T04:27:12.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148539 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T04:27:12.152 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148543 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T04:27:12.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148546 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T04:27:12.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148550 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T04:27:12.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148578 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T04:27:12.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148584 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T04:27:12.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148588 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T04:27:12.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148592 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T04:27:12.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148596 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T04:27:12.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148598 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T04:27:12.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148601 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T04:27:12.156 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148605 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T04:27:12.156 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148607 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T04:27:12.156 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148611 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T04:27:12.156 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148614 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T04:27:12.156 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148618 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T04:27:12.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.148621 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T04:27:12.238 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.242578 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T04:27:12.246 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.252249 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:27:12.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.252273 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:27:12.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.252277 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:27:12.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.252279 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:27:12.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.252282 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:27:12.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.252285 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:27:12.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.252288 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T04:27:12.939 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.944395 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:27:12.940 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.944428 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:27:12.940 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:12.944437 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T04:27:13.097 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.102488 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T04:27:13.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.396250 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:27:13.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.396278 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:27:13.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.396286 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T04:27:13.825 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.830419 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T04:27:13.826 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.830451 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T04:27:13.827 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.830470 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T04:27:13.958 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.963584 I | op-mon: parsing mon endpoints: a=10.104.177.194:6789,b=10.99.92.113:6789,c=10.110.14.128:6789 2022-03-08T04:27:13.963 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.964520 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:27:13.963 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.964557 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:27:13.963 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:13.964572 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T04:27:14.137 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.143187 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:27:14.138 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.143497 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:27:14.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.310909 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T04:27:14.508 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.513974 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T04:27:14.516 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.522275 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T04:27:14.516 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.522297 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T04:27:14.521 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.527057 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T04:27:14.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.570938 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T04:27:14.696 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.698798 I | op-mon: start running mons 2022-03-08T04:27:14.814 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.819415 I | op-mon: parsing mon endpoints: a=10.104.177.194:6789,b=10.99.92.113:6789,c=10.110.14.128:6789 2022-03-08T04:27:14.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:14.959540 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.92.113:6789","10.110.14.128:6789","10.104.177.194:6789"]}] data:a=10.104.177.194:6789,b=10.99.92.113:6789,c=10.110.14.128:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:2] 2022-03-08T04:27:15.525 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:15.530938 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:27:15.526 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:15.531198 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:27:16.518 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:27:17.108 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:27:17.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:17.133975 I | op-mon: targeting the mon count 3 2022-03-08T04:27:17.129 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T04:27:17.130 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T04:27:17.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:17.138074 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:27:17.542 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:17.547865 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:27:17.542 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:17.547892 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:27:17.887 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:17.891821 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:27:17.887 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:17.891854 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:27:18.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:18.242744 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:27:18.238 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:18.242777 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:27:18.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:18.766367 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:27:18.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:18.766395 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T04:27:19.098 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:19.104327 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T04:27:19.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:19.104368 I | op-mon: checking for basic quorum with existing mons 2022-03-08T04:27:19.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:19.220474 I | op-mon: mon "c" endpoint is [v2:10.110.14.128:3300,v1:10.110.14.128:6789] 2022-03-08T04:27:19.289 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:19.295195 I | op-mon: mon "a" endpoint is [v2:10.104.177.194:3300,v1:10.104.177.194:6789] 2022-03-08T04:27:19.528 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:19.533972 I | op-mon: mon "b" endpoint is [v2:10.99.92.113:3300,v1:10.99.92.113:6789] 2022-03-08T04:27:20.128 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:20.134404 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.104.177.194:6789","10.99.92.113:6789","10.110.14.128:6789"]}] data:c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:2] 2022-03-08T04:27:20.735 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:20.740562 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:27:20.735 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:20.740889 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:27:21.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:21.151714 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T04:27:21.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:21.159637 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T04:27:21.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:21.159660 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T04:27:21.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:21.777531 I | op-mon: mons running: [c a b] 2022-03-08T04:27:22.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.193387 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:27:22.194 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.200791 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T04:27:22.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.210724 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T04:27:22.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.210750 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T04:27:22.333 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.338397 I | op-mon: mons running: [c a b] 2022-03-08T04:27:22.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.746200 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:27:22.747 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.752768 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T04:27:22.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.764278 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T04:27:22.759 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.764307 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T04:27:22.932 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:22.937925 I | op-mon: mons running: [c a b] 2022-03-08T04:27:23.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:23.404720 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:27:23.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:23.404749 I | op-mon: mons created: 3 2022-03-08T04:27:23.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:23.800405 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T04:27:23.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:23.822296 I | op-mon: mons running: [c a b] 2022-03-08T04:27:24.225 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:24.230185 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:27:24.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:24.230230 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T04:27:24.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:24.627219 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T04:27:25.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:25.041922 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T04:27:25.550 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:25.555395 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T04:27:26.004 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:26.009580 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T04:27:26.005 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:26.009612 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T04:27:26.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:26.434798 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T04:27:26.917 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:26.921503 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T04:27:26.917 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:26.921526 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T04:27:26.918 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:26.921530 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T04:27:27.131 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:27:27.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:27.351099 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T04:27:27.380 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:27.385061 I | op-mgr: start running mgr 2022-03-08T04:27:27.380 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:27.385106 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T04:27:27.806 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:27:27.832 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T04:27:27.833 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T04:27:27.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:27.883834 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T04:27:27.889 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:27.894030 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T04:27:27.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:27.897696 I | op-mgr: setting services to point to mgr "a" 2022-03-08T04:27:28.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.036427 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T04:27:28.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.036549 I | op-mgr: successful modules: balancer 2022-03-08T04:27:28.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.052620 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T04:27:28.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.052655 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T04:27:28.073 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.077803 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T04:27:28.199 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.204450 I | op-osd: creating 1 new PVCs for device set "smithi018-lv-1" 2022-03-08T04:27:28.381 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.385945 I | op-osd: successfully provisioned PVC "smithi018-lv-1-data-09gnbf" 2022-03-08T04:27:28.381 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.385974 I | op-osd: creating 1 new PVCs for device set "smithi018-lv-2" 2022-03-08T04:27:28.600 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.604748 I | op-osd: successfully provisioned PVC "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:27:28.600 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.604787 I | op-osd: creating 1 new PVCs for device set "smithi018-lv-3" 2022-03-08T04:27:28.779 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.783681 I | op-osd: successfully provisioned PVC "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:27:28.779 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.783704 I | op-osd: creating 1 new PVCs for device set "smithi018-lv-4" 2022-03-08T04:27:28.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.983736 I | op-osd: successfully provisioned PVC "smithi018-lv-4-data-0srk9j" 2022-03-08T04:27:28.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:28.983767 I | op-osd: creating 1 new PVCs for device set "smithi084-lv-1" 2022-03-08T04:27:29.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.124361 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T04:27:29.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.124393 I | op-mgr: successful modules: dashboard 2022-03-08T04:27:29.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.183717 I | op-osd: successfully provisioned PVC "smithi084-lv-1-data-0jzx5v" 2022-03-08T04:27:29.180 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.183758 I | op-osd: creating 1 new PVCs for device set "smithi084-lv-2" 2022-03-08T04:27:29.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.193222 I | op-mgr: successful modules: prometheus 2022-03-08T04:27:29.399 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.403871 I | op-osd: successfully provisioned PVC "smithi084-lv-2-data-0t44bn" 2022-03-08T04:27:29.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.403904 I | op-osd: creating 1 new PVCs for device set "smithi084-lv-3" 2022-03-08T04:27:29.580 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.584220 I | op-osd: successfully provisioned PVC "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:27:29.580 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.584255 I | op-osd: creating 1 new PVCs for device set "smithi084-lv-4" 2022-03-08T04:27:29.780 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.784283 I | op-osd: successfully provisioned PVC "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:27:29.780 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.784320 I | op-osd: creating 1 new PVCs for device set "smithi105-lv-1" 2022-03-08T04:27:29.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.995512 I | op-osd: successfully provisioned PVC "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:27:29.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:29.995553 I | op-osd: creating 1 new PVCs for device set "smithi105-lv-2" 2022-03-08T04:27:30.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:30.183370 I | op-osd: successfully provisioned PVC "smithi105-lv-2-data-04z6t2" 2022-03-08T04:27:30.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:30.183395 I | op-osd: creating 1 new PVCs for device set "smithi105-lv-3" 2022-03-08T04:27:30.408 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:30.412129 I | op-osd: successfully provisioned PVC "smithi105-lv-3-data-0gqb66" 2022-03-08T04:27:30.408 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:30.412179 I | op-osd: creating 1 new PVCs for device set "smithi105-lv-4" 2022-03-08T04:27:30.581 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:30.585625 I | op-osd: successfully provisioned PVC "smithi105-lv-4-data-066ggl" 2022-03-08T04:27:30.602 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:30.607154 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-09gnbf" 2022-03-08T04:27:30.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:30.772962 I | op-mgr: successful modules: orchestrator modules 2022-03-08T04:27:30.994 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:30.997883 I | op-osd: started OSD provisioning job for PVC "smithi018-lv-1-data-09gnbf" 2022-03-08T04:27:30.994 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:30.997915 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:27:31.463 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:31.467112 I | op-osd: started OSD provisioning job for PVC "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:27:31.463 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:31.467147 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:27:32.412 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:32.416790 I | op-osd: started OSD provisioning job for PVC "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:27:32.413 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:32.416827 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0srk9j" 2022-03-08T04:27:32.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:32.890425 I | op-osd: started OSD provisioning job for PVC "smithi018-lv-4-data-0srk9j" 2022-03-08T04:27:32.887 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:32.890459 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-1-data-0jzx5v" 2022-03-08T04:27:33.910 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:33.913982 I | op-osd: started OSD provisioning job for PVC "smithi084-lv-1-data-0jzx5v" 2022-03-08T04:27:33.910 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:33.914018 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-2-data-0t44bn" 2022-03-08T04:27:35.210 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:35.213133 I | op-osd: started OSD provisioning job for PVC "smithi084-lv-2-data-0t44bn" 2022-03-08T04:27:35.210 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:35.213183 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:27:36.057 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:36.060528 I | op-osd: started OSD provisioning job for PVC "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:27:36.057 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:36.060565 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:27:37.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:37.132312 I | op-osd: started OSD provisioning job for PVC "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:27:37.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:37.132336 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:27:37.634 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:37.637454 I | op-osd: started OSD provisioning job for PVC "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:27:37.634 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:37.637479 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-2-data-04z6t2" 2022-03-08T04:27:37.834 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:27:38.058 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.061948 I | op-osd: started OSD provisioning job for PVC "smithi105-lv-2-data-04z6t2" 2022-03-08T04:27:38.059 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.061973 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-3-data-0gqb66" 2022-03-08T04:27:38.288 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.289582 I | op-osd: started OSD provisioning job for PVC "smithi105-lv-3-data-0gqb66" 2022-03-08T04:27:38.288 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.289602 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-4-data-066ggl" 2022-03-08T04:27:38.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.481571 I | op-osd: started OSD provisioning job for PVC "smithi105-lv-4-data-066ggl" 2022-03-08T04:27:38.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.481591 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T04:27:38.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.481594 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T04:27:38.610 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612212 I | op-osd: OSD orchestration status for PVC smithi018-lv-1-data-09gnbf is "starting" 2022-03-08T04:27:38.611 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612261 I | op-osd: OSD orchestration status for PVC smithi018-lv-2-data-0rv5c5 is "starting" 2022-03-08T04:27:38.611 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612284 I | op-osd: OSD orchestration status for PVC smithi018-lv-3-data-0fzdgm is "starting" 2022-03-08T04:27:38.611 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612316 I | op-osd: OSD orchestration status for PVC smithi018-lv-4-data-0srk9j is "starting" 2022-03-08T04:27:38.611 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612345 I | op-osd: OSD orchestration status for PVC smithi084-lv-1-data-0jzx5v is "starting" 2022-03-08T04:27:38.612 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612368 I | op-osd: OSD orchestration status for PVC smithi084-lv-2-data-0t44bn is "starting" 2022-03-08T04:27:38.612 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612390 I | op-osd: OSD orchestration status for PVC smithi084-lv-3-data-0lnxp4 is "starting" 2022-03-08T04:27:38.612 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612414 I | op-osd: OSD orchestration status for PVC smithi084-lv-4-data-0pxmvd is "starting" 2022-03-08T04:27:38.612 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612432 I | op-osd: OSD orchestration status for PVC smithi105-lv-1-data-0pnkl2 is "starting" 2022-03-08T04:27:38.613 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612450 I | op-osd: OSD orchestration status for PVC smithi105-lv-2-data-04z6t2 is "starting" 2022-03-08T04:27:38.613 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612475 I | op-osd: OSD orchestration status for PVC smithi105-lv-3-data-0gqb66 is "starting" 2022-03-08T04:27:38.613 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:38.612496 I | op-osd: OSD orchestration status for PVC smithi105-lv-4-data-066ggl is "starting" 2022-03-08T04:27:39.228 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:27:39.252 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T04:27:39.252 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T04:27:49.254 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:27:50.804 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:27:50.831 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-08T04:27:50.831 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T04:27:51.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:51.644843 I | op-osd: OSD orchestration status for node smithi084-lv-1-data-0jzx5v is "orchestrating" 2022-03-08T04:27:52.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:52.799805 I | op-osd: OSD orchestration status for PVC smithi084-lv-1-data-0jzx5v is "orchestrating" 2022-03-08T04:27:55.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:55.052350 I | op-osd: OSD orchestration status for node smithi084-lv-2-data-0t44bn is "orchestrating" 2022-03-08T04:27:56.399 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:56.400988 I | op-osd: OSD orchestration status for PVC smithi084-lv-2-data-0t44bn is "orchestrating" 2022-03-08T04:27:56.949 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:56.950859 I | op-osd: OSD orchestration status for node smithi084-lv-4-data-0pxmvd is "orchestrating" 2022-03-08T04:27:56.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:56.967070 I | op-osd: OSD orchestration status for node smithi018-lv-2-data-0rv5c5 is "orchestrating" 2022-03-08T04:27:57.111 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:57.113526 I | op-osd: OSD orchestration status for node smithi018-lv-1-data-09gnbf is "orchestrating" 2022-03-08T04:27:57.452 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:57.454852 I | op-osd: OSD orchestration status for PVC smithi084-lv-4-data-0pxmvd is "orchestrating" 2022-03-08T04:27:57.473 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:57.475539 I | op-osd: OSD orchestration status for node smithi018-lv-3-data-0fzdgm is "orchestrating" 2022-03-08T04:27:57.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:57.501282 I | op-osd: OSD orchestration status for node smithi018-lv-4-data-0srk9j is "orchestrating" 2022-03-08T04:27:57.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:57.675811 I | op-osd: OSD orchestration status for PVC smithi018-lv-2-data-0rv5c5 is "orchestrating" 2022-03-08T04:27:57.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:57.683605 I | op-osd: OSD orchestration status for PVC smithi018-lv-1-data-09gnbf is "orchestrating" 2022-03-08T04:27:57.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:57.762052 I | op-osd: OSD orchestration status for node smithi084-lv-3-data-0lnxp4 is "orchestrating" 2022-03-08T04:27:57.917 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:57.918630 I | op-osd: OSD orchestration status for PVC smithi018-lv-3-data-0fzdgm is "orchestrating" 2022-03-08T04:27:58.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:58.050054 I | op-osd: OSD orchestration status for PVC smithi018-lv-4-data-0srk9j is "orchestrating" 2022-03-08T04:27:58.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:58.229022 I | op-osd: OSD orchestration status for node smithi105-lv-2-data-04z6t2 is "orchestrating" 2022-03-08T04:27:58.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:58.229105 I | op-osd: OSD orchestration status for node smithi105-lv-1-data-0pnkl2 is "orchestrating" 2022-03-08T04:27:58.231 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:58.233820 I | op-osd: OSD orchestration status for PVC smithi084-lv-3-data-0lnxp4 is "orchestrating" 2022-03-08T04:27:58.698 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:58.699987 I | op-osd: OSD orchestration status for PVC smithi105-lv-2-data-04z6t2 is "orchestrating" 2022-03-08T04:27:58.748 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:27:58.749784 I | op-osd: OSD orchestration status for PVC smithi105-lv-1-data-0pnkl2 is "orchestrating" 2022-03-08T04:28:00.833 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:28:01.400 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:28:01.422 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1646713681,"num_remapped_pgs":0} 2022-03-08T04:28:01.423 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T04:28:01.605 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:01.607142 I | op-osd: OSD orchestration status for node smithi105-lv-3-data-0gqb66 is "orchestrating" 2022-03-08T04:28:02.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:02.019789 I | op-osd: OSD orchestration status for PVC smithi105-lv-3-data-0gqb66 is "orchestrating" 2022-03-08T04:28:02.307 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:02.308866 I | op-osd: OSD orchestration status for PVC smithi084-lv-1-data-0jzx5v is "completed" 2022-03-08T04:28:02.307 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:02.308917 I | op-osd: creating OSD 0 on PVC "smithi084-lv-1-data-0jzx5v" 2022-03-08T04:28:02.308 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:02.308939 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-1-data-0jzx5v" 2022-03-08T04:28:03.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:03.394956 I | op-osd: OSD orchestration status for node smithi105-lv-4-data-066ggl is "orchestrating" 2022-03-08T04:28:03.807 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:03.808976 I | op-osd: OSD orchestration status for PVC smithi105-lv-4-data-066ggl is "orchestrating" 2022-03-08T04:28:04.448 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:04.449876 I | op-osd: OSD orchestration status for PVC smithi018-lv-1-data-09gnbf is "completed" 2022-03-08T04:28:04.448 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:04.449920 I | op-osd: creating OSD 3 on PVC "smithi018-lv-1-data-09gnbf" 2022-03-08T04:28:04.449 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:04.449937 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-09gnbf" 2022-03-08T04:28:05.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:05.502599 I | op-osd: OSD orchestration status for PVC smithi084-lv-2-data-0t44bn is "completed" 2022-03-08T04:28:05.502 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:05.502622 I | op-osd: creating OSD 1 on PVC "smithi084-lv-2-data-0t44bn" 2022-03-08T04:28:05.502 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:05.502631 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-2-data-0t44bn" 2022-03-08T04:28:06.698 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:06.698784 I | op-osd: OSD orchestration status for PVC smithi084-lv-4-data-0pxmvd is "completed" 2022-03-08T04:28:06.699 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:06.698844 I | op-osd: creating OSD 2 on PVC "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:28:06.699 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:06.698870 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:28:07.137 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:07.139130 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi018": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:28:09.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:09.265663 I | op-osd: OSD orchestration status for PVC smithi018-lv-2-data-0rv5c5 is "completed" 2022-03-08T04:28:09.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:09.265699 I | op-osd: creating OSD 4 on PVC "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:28:09.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:09.265712 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:28:10.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:10.623901 I | op-osd: OSD orchestration status for PVC smithi105-lv-2-data-04z6t2 is "completed" 2022-03-08T04:28:10.628 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:10.623949 I | op-osd: creating OSD 8 on PVC "smithi105-lv-2-data-04z6t2" 2022-03-08T04:28:10.628 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:10.623975 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-2-data-04z6t2" 2022-03-08T04:28:11.424 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:28:11.584 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:11.585220 I | op-osd: OSD orchestration status for PVC smithi018-lv-3-data-0fzdgm is "completed" 2022-03-08T04:28:11.585 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:11.585274 I | op-osd: creating OSD 7 on PVC "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:28:11.585 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:11.585286 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:28:12.763 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:28:12.786 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646713685,"num_remapped_pgs":0} 2022-03-08T04:28:12.786 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T04:28:13.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:13.228533 I | op-osd: OSD orchestration status for PVC smithi018-lv-4-data-0srk9j is "completed" 2022-03-08T04:28:13.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:13.228571 I | op-osd: creating OSD 6 on PVC "smithi018-lv-4-data-0srk9j" 2022-03-08T04:28:13.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:13.228592 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0srk9j" 2022-03-08T04:28:14.587 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:14.588662 I | op-osd: OSD orchestration status for PVC smithi084-lv-3-data-0lnxp4 is "completed" 2022-03-08T04:28:14.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:14.588735 I | op-osd: creating OSD 5 on PVC "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:28:14.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:14.588746 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:28:16.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:16.456437 I | op-osd: OSD orchestration status for PVC smithi105-lv-1-data-0pnkl2 is "completed" 2022-03-08T04:28:16.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:16.456463 I | op-osd: creating OSD 9 on PVC "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:28:16.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:16.456475 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:28:17.752 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:17.733099 I | op-osd: OSD orchestration status for PVC smithi105-lv-3-data-0gqb66 is "completed" 2022-03-08T04:28:17.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:17.733153 I | op-osd: creating OSD 10 on PVC "smithi105-lv-3-data-0gqb66" 2022-03-08T04:28:17.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:17.733179 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-3-data-0gqb66" 2022-03-08T04:28:19.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:19.019899 I | op-osd: OSD orchestration status for PVC smithi105-lv-4-data-066ggl is "completed" 2022-03-08T04:28:19.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:19.019926 I | op-osd: creating OSD 11 on PVC "smithi105-lv-4-data-066ggl" 2022-03-08T04:28:19.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:19.019954 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-4-data-066ggl" 2022-03-08T04:28:21.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:21.679448 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T04:28:21.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:28:21.679470 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T04:28:22.789 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:28:24.971 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:28:24.994 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646713685,"num_remapped_pgs":0} 2022-03-08T04:28:24.995 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T04:28:34.996 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:28:36.172 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:28:36.192 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646713685,"num_remapped_pgs":0} 2022-03-08T04:28:36.193 INFO:tasks.rook: have 0/12 OSDs 2022-03-08T04:28:46.194 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:28:47.358 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:28:47.379 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":19,"num_osds":12,"num_up_osds":2,"osd_up_since":1646713725,"num_in_osds":12,"osd_in_since":1646713685,"num_remapped_pgs":0} 2022-03-08T04:28:47.380 INFO:tasks.rook: have 2/12 OSDs 2022-03-08T04:28:57.380 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:28:57.935 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:28:57.954 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":26,"num_osds":12,"num_up_osds":9,"osd_up_since":1646713736,"num_in_osds":12,"osd_in_since":1646713685,"num_remapped_pgs":1} 2022-03-08T04:28:57.955 INFO:tasks.rook: have 9/12 OSDs 2022-03-08T04:29:07.956 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd stat -f json 2022-03-08T04:29:08.503 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:08.522 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":30,"num_osds":12,"num_up_osds":12,"osd_up_since":1646713739,"num_in_osds":12,"osd_in_since":1646713685,"num_remapped_pgs":0} 2022-03-08T04:29:08.522 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-08T04:29:08.523 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- cat /etc/ceph/ceph.conf 2022-03-08T04:29:08.726 INFO:teuthology.orchestra.run.smithi018.stdout:[global] 2022-03-08T04:29:08.726 INFO:teuthology.orchestra.run.smithi018.stdout:mon_host = 10.110.14.128:6789,10.104.177.194:6789,10.99.92.113:6789 2022-03-08T04:29:08.726 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:08.727 INFO:teuthology.orchestra.run.smithi018.stdout:[client.admin] 2022-03-08T04:29:08.727 INFO:teuthology.orchestra.run.smithi018.stdout:keyring = /etc/ceph/keyring 2022-03-08T04:29:08.730 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- cat /etc/ceph/keyring 2022-03-08T04:29:08.926 INFO:teuthology.orchestra.run.smithi018.stdout:[client.admin] 2022-03-08T04:29:08.927 INFO:teuthology.orchestra.run.smithi018.stdout:key = AQBc2iZiabZyNxAANanXY6SrHF6y9VSwulSpyQ== 2022-03-08T04:29:08.931 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /etc/ceph 2022-03-08T04:29:08.945 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph 2022-03-08T04:29:08.962 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2022-03-08T04:29:08.978 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:29:08.979 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T04:29:08.992 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:29:08.993 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/keyring 2022-03-08T04:29:09.004 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:29:09.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T04:29:09.019 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:29:09.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/keyring 2022-03-08T04:29:09.077 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:29:09.078 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2022-03-08T04:29:09.092 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:29:09.093 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/keyring 2022-03-08T04:29:09.106 INFO:tasks.rook:Setting up client nodes... 2022-03-08T04:29:09.107 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T04:29:10.023 INFO:teuthology.orchestra.run.smithi018.stdout:[client.a] 2022-03-08T04:29:10.024 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCV2yZimYuUMBAA0+OWV/Z5UPlZMOOjlq84kA== 2022-03-08T04:29:10.049 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-08T04:29:10.050 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-08T04:29:10.050 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-08T04:29:10.075 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T04:29:10.941 INFO:teuthology.orchestra.run.smithi018.stdout:[client.b] 2022-03-08T04:29:10.942 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCW2yZiJX7SKBAA/zmGProJmjhZmkkcrSr9Jw== 2022-03-08T04:29:10.965 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2022-03-08T04:29:10.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-08T04:29:10.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-08T04:29:10.981 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-08T04:29:11.785 INFO:teuthology.orchestra.run.smithi018.stdout:[client.c] 2022-03-08T04:29:11.786 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCX2yZiqGgnJxAAXE8CiN5635prthCb0D+yLw== 2022-03-08T04:29:11.809 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2022-03-08T04:29:11.810 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-08T04:29:11.810 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-08T04:29:11.830 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd dump --format=json 2022-03-08T04:29:12.367 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:12.368 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":30,"fsid":"85af3f2f-7009-488d-b93d-f17d85a9eb4e","created":"2022-03-08T04:24:22.705128+0000","modified":"2022-03-08T04:29:01.351530+0000","last_up_change":"2022-03-08T04:28:59.064370+0000","last_in_change":"2022-03-08T04:28:05.143581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T04:28:51.227794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d37d33c5-8ba5-48fc-b589-5b1183f79ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6800","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6801","nonce":3878686972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6802","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6803","nonce":3878686972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6806","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6807","nonce":3878686972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6804","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6805","nonce":3878686972}]},"public_addr":"10.248.137.79:6801/3878686972","cluster_addr":"10.248.137.79:6803/3878686972","heartbeat_back_addr":"10.248.137.79:6807/3878686972","heartbeat_front_addr":"10.248.137.79:6805/3878686972","state":["exists","up"]},{"osd":1,"uuid":"d0b87b82-4799-4f84-b81d-33407a28cb70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6800","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6801","nonce":154273419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6802","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6803","nonce":154273419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6806","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6807","nonce":154273419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6804","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6805","nonce":154273419}]},"public_addr":"10.248.137.80:6801/154273419","cluster_addr":"10.248.137.80:6803/154273419","heartbeat_back_addr":"10.248.137.80:6807/154273419","heartbeat_front_addr":"10.248.137.80:6805/154273419","state":["exists","up"]},{"osd":2,"uuid":"e355f862-75e2-40c1-a286-28192441b314","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6800","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6801","nonce":1932546855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6802","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6803","nonce":1932546855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6806","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6807","nonce":1932546855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6804","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6805","nonce":1932546855}]},"public_addr":"10.248.137.81:6801/1932546855","cluster_addr":"10.248.137.81:6803/1932546855","heartbeat_back_addr":"10.248.137.81:6807/1932546855","heartbeat_front_addr":"10.248.137.81:6805/1932546855","state":["exists","up"]},{"osd":3,"uuid":"7e01fb62-aa8a-4f3c-b5af-67ee9bb8602d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6800","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6801","nonce":2408150371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6802","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6803","nonce":2408150371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6806","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6807","nonce":2408150371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6804","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6805","nonce":2408150371}]},"public_addr":"10.248.138.75:6801/2408150371","cluster_addr":"10.248.138.75:6803/2408150371","heartbeat_back_addr":"10.248.138.75:6807/2408150371","heartbeat_front_addr":"10.248.138.75:6805/2408150371","state":["exists","up"]},{"osd":4,"uuid":"8b4a5ed4-fb5e-4c4d-b642-17904b45a860","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6800","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6801","nonce":1194817709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6802","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6803","nonce":1194817709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6806","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6807","nonce":1194817709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6804","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6805","nonce":1194817709}]},"public_addr":"10.248.138.77:6801/1194817709","cluster_addr":"10.248.138.77:6803/1194817709","heartbeat_back_addr":"10.248.138.77:6807/1194817709","heartbeat_front_addr":"10.248.138.77:6805/1194817709","state":["exists","up"]},{"osd":5,"uuid":"41d1fa9a-81b3-43ac-bd92-7b1445490847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6800","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6801","nonce":508452244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6802","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6803","nonce":508452244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6806","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6807","nonce":508452244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6804","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6805","nonce":508452244}]},"public_addr":"10.248.137.82:6801/508452244","cluster_addr":"10.248.137.82:6803/508452244","heartbeat_back_addr":"10.248.137.82:6807/508452244","heartbeat_front_addr":"10.248.137.82:6805/508452244","state":["exists","up"]},{"osd":6,"uuid":"1ee44b22-b3f3-4b49-baca-e77216c2cf56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6800","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6801","nonce":577153597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6802","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6803","nonce":577153597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6806","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6807","nonce":577153597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6804","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6805","nonce":577153597}]},"public_addr":"10.248.138.79:6801/577153597","cluster_addr":"10.248.138.79:6803/577153597","heartbeat_back_addr":"10.248.138.79:6807/577153597","heartbeat_front_addr":"10.248.138.79:6805/577153597","state":["exists","up"]},{"osd":7,"uuid":"d120cec7-b9f0-43b6-ad2f-098161e75cc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6800","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6801","nonce":1360075654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6802","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6803","nonce":1360075654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6806","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6807","nonce":1360075654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6804","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6805","nonce":1360075654}]},"public_addr":"10.248.138.78:6801/1360075654","cluster_addr":"10.248.138.78:6803/1360075654","heartbeat_back_addr":"10.248.138.78:6807/1360075654","heartbeat_front_addr":"10.248.138.78:6805/1360075654","state":["exists","up"]},{"osd":8,"uuid":"575e3221-5e28-45b9-98ec-6e2380455285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6800","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6801","nonce":2256135522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6802","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6803","nonce":2256135522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6806","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6807","nonce":2256135522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6804","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6805","nonce":2256135522}]},"public_addr":"10.248.141.12:6801/2256135522","cluster_addr":"10.248.141.12:6803/2256135522","heartbeat_back_addr":"10.248.141.12:6807/2256135522","heartbeat_front_addr":"10.248.141.12:6805/2256135522","state":["exists","up"]},{"osd":9,"uuid":"9c0159c8-2b85-4c74-83d2-c84656dae52b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6800","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6801","nonce":1297228769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6802","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6803","nonce":1297228769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6806","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6807","nonce":1297228769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6804","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6805","nonce":1297228769}]},"public_addr":"10.248.141.13:6801/1297228769","cluster_addr":"10.248.141.13:6803/1297228769","heartbeat_back_addr":"10.248.141.13:6807/1297228769","heartbeat_front_addr":"10.248.141.13:6805/1297228769","state":["exists","up"]},{"osd":10,"uuid":"5b3ff147-83d7-48cd-93ac-0aa8fb870644","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6800","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6801","nonce":1868757818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6802","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6803","nonce":1868757818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6806","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6807","nonce":1868757818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6804","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6805","nonce":1868757818}]},"public_addr":"10.248.141.14:6801/1868757818","cluster_addr":"10.248.141.14:6803/1868757818","heartbeat_back_addr":"10.248.141.14:6807/1868757818","heartbeat_front_addr":"10.248.141.14:6805/1868757818","state":["exists","up"]},{"osd":11,"uuid":"ba9f3fd9-2400-4b19-a833-d8a567c2dcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6800","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6801","nonce":3429103514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6802","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6803","nonce":3429103514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6806","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6807","nonce":3429103514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6804","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6805","nonce":3429103514}]},"public_addr":"10.248.141.15:6801/3429103514","cluster_addr":"10.248.141.15:6803/3429103514","heartbeat_back_addr":"10.248.141.15:6807/3429103514","heartbeat_front_addr":"10.248.141.15:6805/3429103514","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:35.312051+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:42.002514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:44.777852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:55.340792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:57.156252+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:45.315260+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:55.962301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:57.062952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:40.526755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:47.552165+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:49.381787+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:53.441578+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.138.69:6801/3230304984":"2022-03-09T04:27:02.360983+0000","10.248.138.69:0/514887801":"2022-03-09T04:27:02.360983+0000","10.248.138.69:6800/3230304984":"2022-03-09T04:27:02.360983+0000","10.248.138.69:0/3001458541":"2022-03-09T04:27:02.360983+0000","10.248.138.69:0/1067168024":"2022-03-09T04:27:02.360983+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T04:29:12.385 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-08T04:28:51.227794+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-08T04:29:12.386 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd pool get .mgr pg_num 2022-03-08T04:29:12.931 INFO:teuthology.orchestra.run.smithi018.stdout:pg_num: 1 2022-03-08T04:29:12.968 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-08T04:29:12.969 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-08T04:29:12.969 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph mgr dump --format=json 2022-03-08T04:29:13.538 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:13.557 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":13,"active_gid":14281,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.69:6800","nonce":2830146802},{"type":"v1","addr":"10.248.138.69:6801","nonce":2830146802}]},"active_addr":"10.248.138.69:6801/2830146802","active_change":"2022-03-08T04:27:02.385401+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.248.138.69:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.248.138.69:0","nonce":1062255349}]},{"addrvec":[{"type":"v2","addr":"10.248.138.69:0","nonce":3054264457}]},{"addrvec":[{"type":"v2","addr":"10.248.138.69:0","nonce":1027336206}]},{"addrvec":[{"type":"v2","addr":"10.248.138.69:0","nonce":1084437468}]}]}} 2022-03-08T04:29:13.561 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-08T04:29:13.562 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-08T04:29:13.562 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd dump --format=json 2022-03-08T04:29:14.125 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:14.125 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":30,"fsid":"85af3f2f-7009-488d-b93d-f17d85a9eb4e","created":"2022-03-08T04:24:22.705128+0000","modified":"2022-03-08T04:29:01.351530+0000","last_up_change":"2022-03-08T04:28:59.064370+0000","last_in_change":"2022-03-08T04:28:05.143581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T04:28:51.227794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d37d33c5-8ba5-48fc-b589-5b1183f79ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6800","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6801","nonce":3878686972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6802","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6803","nonce":3878686972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6806","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6807","nonce":3878686972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6804","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6805","nonce":3878686972}]},"public_addr":"10.248.137.79:6801/3878686972","cluster_addr":"10.248.137.79:6803/3878686972","heartbeat_back_addr":"10.248.137.79:6807/3878686972","heartbeat_front_addr":"10.248.137.79:6805/3878686972","state":["exists","up"]},{"osd":1,"uuid":"d0b87b82-4799-4f84-b81d-33407a28cb70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6800","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6801","nonce":154273419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6802","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6803","nonce":154273419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6806","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6807","nonce":154273419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6804","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6805","nonce":154273419}]},"public_addr":"10.248.137.80:6801/154273419","cluster_addr":"10.248.137.80:6803/154273419","heartbeat_back_addr":"10.248.137.80:6807/154273419","heartbeat_front_addr":"10.248.137.80:6805/154273419","state":["exists","up"]},{"osd":2,"uuid":"e355f862-75e2-40c1-a286-28192441b314","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6800","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6801","nonce":1932546855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6802","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6803","nonce":1932546855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6806","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6807","nonce":1932546855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6804","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6805","nonce":1932546855}]},"public_addr":"10.248.137.81:6801/1932546855","cluster_addr":"10.248.137.81:6803/1932546855","heartbeat_back_addr":"10.248.137.81:6807/1932546855","heartbeat_front_addr":"10.248.137.81:6805/1932546855","state":["exists","up"]},{"osd":3,"uuid":"7e01fb62-aa8a-4f3c-b5af-67ee9bb8602d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6800","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6801","nonce":2408150371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6802","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6803","nonce":2408150371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6806","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6807","nonce":2408150371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6804","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6805","nonce":2408150371}]},"public_addr":"10.248.138.75:6801/2408150371","cluster_addr":"10.248.138.75:6803/2408150371","heartbeat_back_addr":"10.248.138.75:6807/2408150371","heartbeat_front_addr":"10.248.138.75:6805/2408150371","state":["exists","up"]},{"osd":4,"uuid":"8b4a5ed4-fb5e-4c4d-b642-17904b45a860","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6800","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6801","nonce":1194817709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6802","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6803","nonce":1194817709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6806","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6807","nonce":1194817709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6804","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6805","nonce":1194817709}]},"public_addr":"10.248.138.77:6801/1194817709","cluster_addr":"10.248.138.77:6803/1194817709","heartbeat_back_addr":"10.248.138.77:6807/1194817709","heartbeat_front_addr":"10.248.138.77:6805/1194817709","state":["exists","up"]},{"osd":5,"uuid":"41d1fa9a-81b3-43ac-bd92-7b1445490847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6800","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6801","nonce":508452244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6802","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6803","nonce":508452244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6806","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6807","nonce":508452244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6804","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6805","nonce":508452244}]},"public_addr":"10.248.137.82:6801/508452244","cluster_addr":"10.248.137.82:6803/508452244","heartbeat_back_addr":"10.248.137.82:6807/508452244","heartbeat_front_addr":"10.248.137.82:6805/508452244","state":["exists","up"]},{"osd":6,"uuid":"1ee44b22-b3f3-4b49-baca-e77216c2cf56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6800","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6801","nonce":577153597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6802","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6803","nonce":577153597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6806","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6807","nonce":577153597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6804","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6805","nonce":577153597}]},"public_addr":"10.248.138.79:6801/577153597","cluster_addr":"10.248.138.79:6803/577153597","heartbeat_back_addr":"10.248.138.79:6807/577153597","heartbeat_front_addr":"10.248.138.79:6805/577153597","state":["exists","up"]},{"osd":7,"uuid":"d120cec7-b9f0-43b6-ad2f-098161e75cc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6800","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6801","nonce":1360075654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6802","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6803","nonce":1360075654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6806","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6807","nonce":1360075654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6804","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6805","nonce":1360075654}]},"public_addr":"10.248.138.78:6801/1360075654","cluster_addr":"10.248.138.78:6803/1360075654","heartbeat_back_addr":"10.248.138.78:6807/1360075654","heartbeat_front_addr":"10.248.138.78:6805/1360075654","state":["exists","up"]},{"osd":8,"uuid":"575e3221-5e28-45b9-98ec-6e2380455285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6800","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6801","nonce":2256135522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6802","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6803","nonce":2256135522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6806","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6807","nonce":2256135522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6804","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6805","nonce":2256135522}]},"public_addr":"10.248.141.12:6801/2256135522","cluster_addr":"10.248.141.12:6803/2256135522","heartbeat_back_addr":"10.248.141.12:6807/2256135522","heartbeat_front_addr":"10.248.141.12:6805/2256135522","state":["exists","up"]},{"osd":9,"uuid":"9c0159c8-2b85-4c74-83d2-c84656dae52b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6800","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6801","nonce":1297228769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6802","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6803","nonce":1297228769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6806","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6807","nonce":1297228769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6804","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6805","nonce":1297228769}]},"public_addr":"10.248.141.13:6801/1297228769","cluster_addr":"10.248.141.13:6803/1297228769","heartbeat_back_addr":"10.248.141.13:6807/1297228769","heartbeat_front_addr":"10.248.141.13:6805/1297228769","state":["exists","up"]},{"osd":10,"uuid":"5b3ff147-83d7-48cd-93ac-0aa8fb870644","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6800","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6801","nonce":1868757818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6802","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6803","nonce":1868757818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6806","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6807","nonce":1868757818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6804","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6805","nonce":1868757818}]},"public_addr":"10.248.141.14:6801/1868757818","cluster_addr":"10.248.141.14:6803/1868757818","heartbeat_back_addr":"10.248.141.14:6807/1868757818","heartbeat_front_addr":"10.248.141.14:6805/1868757818","state":["exists","up"]},{"osd":11,"uuid":"ba9f3fd9-2400-4b19-a833-d8a567c2dcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6800","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6801","nonce":3429103514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6802","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6803","nonce":3429103514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6806","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6807","nonce":3429103514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6804","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6805","nonce":3429103514}]},"public_addr":"10.248.141.15:6801/3429103514","cluster_addr":"10.248.141.15:6803/3429103514","heartbeat_back_addr":"10.248.141.15:6807/3429103514","heartbeat_front_addr":"10.248.141.15:6805/3429103514","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:35.312051+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:42.002514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:44.777852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:55.340792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:57.156252+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:45.315260+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:55.962301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:57.062952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:40.526755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:47.552165+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:49.381787+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:53.441578+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.138.69:6801/3230304984":"2022-03-09T04:27:02.360983+0000","10.248.138.69:0/514887801":"2022-03-09T04:27:02.360983+0000","10.248.138.69:6800/3230304984":"2022-03-09T04:27:02.360983+0000","10.248.138.69:0/3001458541":"2022-03-09T04:27:02.360983+0000","10.248.138.69:0/1067168024":"2022-03-09T04:27:02.360983+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T04:29:14.144 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-08T04:29:14.145 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd dump --format=json 2022-03-08T04:29:14.669 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:14.670 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":30,"fsid":"85af3f2f-7009-488d-b93d-f17d85a9eb4e","created":"2022-03-08T04:24:22.705128+0000","modified":"2022-03-08T04:29:01.351530+0000","last_up_change":"2022-03-08T04:28:59.064370+0000","last_in_change":"2022-03-08T04:28:05.143581+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-08T04:28:51.227794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d37d33c5-8ba5-48fc-b589-5b1183f79ab2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6800","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6801","nonce":3878686972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6802","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6803","nonce":3878686972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6806","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6807","nonce":3878686972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.79:6804","nonce":3878686972},{"type":"v1","addr":"10.248.137.79:6805","nonce":3878686972}]},"public_addr":"10.248.137.79:6801/3878686972","cluster_addr":"10.248.137.79:6803/3878686972","heartbeat_back_addr":"10.248.137.79:6807/3878686972","heartbeat_front_addr":"10.248.137.79:6805/3878686972","state":["exists","up"]},{"osd":1,"uuid":"d0b87b82-4799-4f84-b81d-33407a28cb70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6800","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6801","nonce":154273419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6802","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6803","nonce":154273419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6806","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6807","nonce":154273419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.80:6804","nonce":154273419},{"type":"v1","addr":"10.248.137.80:6805","nonce":154273419}]},"public_addr":"10.248.137.80:6801/154273419","cluster_addr":"10.248.137.80:6803/154273419","heartbeat_back_addr":"10.248.137.80:6807/154273419","heartbeat_front_addr":"10.248.137.80:6805/154273419","state":["exists","up"]},{"osd":2,"uuid":"e355f862-75e2-40c1-a286-28192441b314","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6800","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6801","nonce":1932546855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6802","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6803","nonce":1932546855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6806","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6807","nonce":1932546855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.81:6804","nonce":1932546855},{"type":"v1","addr":"10.248.137.81:6805","nonce":1932546855}]},"public_addr":"10.248.137.81:6801/1932546855","cluster_addr":"10.248.137.81:6803/1932546855","heartbeat_back_addr":"10.248.137.81:6807/1932546855","heartbeat_front_addr":"10.248.137.81:6805/1932546855","state":["exists","up"]},{"osd":3,"uuid":"7e01fb62-aa8a-4f3c-b5af-67ee9bb8602d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6800","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6801","nonce":2408150371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6802","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6803","nonce":2408150371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6806","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6807","nonce":2408150371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.75:6804","nonce":2408150371},{"type":"v1","addr":"10.248.138.75:6805","nonce":2408150371}]},"public_addr":"10.248.138.75:6801/2408150371","cluster_addr":"10.248.138.75:6803/2408150371","heartbeat_back_addr":"10.248.138.75:6807/2408150371","heartbeat_front_addr":"10.248.138.75:6805/2408150371","state":["exists","up"]},{"osd":4,"uuid":"8b4a5ed4-fb5e-4c4d-b642-17904b45a860","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6800","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6801","nonce":1194817709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6802","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6803","nonce":1194817709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6806","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6807","nonce":1194817709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.77:6804","nonce":1194817709},{"type":"v1","addr":"10.248.138.77:6805","nonce":1194817709}]},"public_addr":"10.248.138.77:6801/1194817709","cluster_addr":"10.248.138.77:6803/1194817709","heartbeat_back_addr":"10.248.138.77:6807/1194817709","heartbeat_front_addr":"10.248.138.77:6805/1194817709","state":["exists","up"]},{"osd":5,"uuid":"41d1fa9a-81b3-43ac-bd92-7b1445490847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6800","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6801","nonce":508452244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6802","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6803","nonce":508452244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6806","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6807","nonce":508452244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.137.82:6804","nonce":508452244},{"type":"v1","addr":"10.248.137.82:6805","nonce":508452244}]},"public_addr":"10.248.137.82:6801/508452244","cluster_addr":"10.248.137.82:6803/508452244","heartbeat_back_addr":"10.248.137.82:6807/508452244","heartbeat_front_addr":"10.248.137.82:6805/508452244","state":["exists","up"]},{"osd":6,"uuid":"1ee44b22-b3f3-4b49-baca-e77216c2cf56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6800","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6801","nonce":577153597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6802","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6803","nonce":577153597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6806","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6807","nonce":577153597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.79:6804","nonce":577153597},{"type":"v1","addr":"10.248.138.79:6805","nonce":577153597}]},"public_addr":"10.248.138.79:6801/577153597","cluster_addr":"10.248.138.79:6803/577153597","heartbeat_back_addr":"10.248.138.79:6807/577153597","heartbeat_front_addr":"10.248.138.79:6805/577153597","state":["exists","up"]},{"osd":7,"uuid":"d120cec7-b9f0-43b6-ad2f-098161e75cc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6800","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6801","nonce":1360075654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6802","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6803","nonce":1360075654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6806","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6807","nonce":1360075654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.138.78:6804","nonce":1360075654},{"type":"v1","addr":"10.248.138.78:6805","nonce":1360075654}]},"public_addr":"10.248.138.78:6801/1360075654","cluster_addr":"10.248.138.78:6803/1360075654","heartbeat_back_addr":"10.248.138.78:6807/1360075654","heartbeat_front_addr":"10.248.138.78:6805/1360075654","state":["exists","up"]},{"osd":8,"uuid":"575e3221-5e28-45b9-98ec-6e2380455285","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6800","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6801","nonce":2256135522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6802","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6803","nonce":2256135522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6806","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6807","nonce":2256135522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.12:6804","nonce":2256135522},{"type":"v1","addr":"10.248.141.12:6805","nonce":2256135522}]},"public_addr":"10.248.141.12:6801/2256135522","cluster_addr":"10.248.141.12:6803/2256135522","heartbeat_back_addr":"10.248.141.12:6807/2256135522","heartbeat_front_addr":"10.248.141.12:6805/2256135522","state":["exists","up"]},{"osd":9,"uuid":"9c0159c8-2b85-4c74-83d2-c84656dae52b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6800","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6801","nonce":1297228769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6802","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6803","nonce":1297228769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6806","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6807","nonce":1297228769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.13:6804","nonce":1297228769},{"type":"v1","addr":"10.248.141.13:6805","nonce":1297228769}]},"public_addr":"10.248.141.13:6801/1297228769","cluster_addr":"10.248.141.13:6803/1297228769","heartbeat_back_addr":"10.248.141.13:6807/1297228769","heartbeat_front_addr":"10.248.141.13:6805/1297228769","state":["exists","up"]},{"osd":10,"uuid":"5b3ff147-83d7-48cd-93ac-0aa8fb870644","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6800","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6801","nonce":1868757818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6802","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6803","nonce":1868757818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6806","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6807","nonce":1868757818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.14:6804","nonce":1868757818},{"type":"v1","addr":"10.248.141.14:6805","nonce":1868757818}]},"public_addr":"10.248.141.14:6801/1868757818","cluster_addr":"10.248.141.14:6803/1868757818","heartbeat_back_addr":"10.248.141.14:6807/1868757818","heartbeat_front_addr":"10.248.141.14:6805/1868757818","state":["exists","up"]},{"osd":11,"uuid":"ba9f3fd9-2400-4b19-a833-d8a567c2dcbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6800","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6801","nonce":3429103514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6802","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6803","nonce":3429103514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6806","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6807","nonce":3429103514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.141.15:6804","nonce":3429103514},{"type":"v1","addr":"10.248.141.15:6805","nonce":3429103514}]},"public_addr":"10.248.141.15:6801/3429103514","cluster_addr":"10.248.141.15:6803/3429103514","heartbeat_back_addr":"10.248.141.15:6807/3429103514","heartbeat_front_addr":"10.248.141.15:6805/3429103514","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:35.312051+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:42.002514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:44.777852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:55.340792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:57.156252+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:45.315260+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:55.962301+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:57.062952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:40.526755+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:47.552165+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:49.381787+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-08T04:28:53.441578+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.138.69:6801/3230304984":"2022-03-09T04:27:02.360983+0000","10.248.138.69:0/514887801":"2022-03-09T04:27:02.360983+0000","10.248.138.69:6800/3230304984":"2022-03-09T04:27:02.360983+0000","10.248.138.69:0/3001458541":"2022-03-09T04:27:02.360983+0000","10.248.138.69:0/1067168024":"2022-03-09T04:27:02.360983+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-08T04:29:14.692 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.0 flush_pg_stats 2022-03-08T04:29:14.693 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.1 flush_pg_stats 2022-03-08T04:29:14.694 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.2 flush_pg_stats 2022-03-08T04:29:14.694 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.3 flush_pg_stats 2022-03-08T04:29:14.695 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.4 flush_pg_stats 2022-03-08T04:29:14.696 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.5 flush_pg_stats 2022-03-08T04:29:14.696 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.6 flush_pg_stats 2022-03-08T04:29:14.697 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.7 flush_pg_stats 2022-03-08T04:29:14.698 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.8 flush_pg_stats 2022-03-08T04:29:14.698 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.9 flush_pg_stats 2022-03-08T04:29:14.699 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.10 flush_pg_stats 2022-03-08T04:29:14.699 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph tell osd.11 flush_pg_stats 2022-03-08T04:29:15.226 INFO:teuthology.orchestra.run.smithi018.stdout:90194313225 2022-03-08T04:29:15.227 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.2 2022-03-08T04:29:15.443 INFO:teuthology.orchestra.run.smithi018.stdout:73014444041 2022-03-08T04:29:15.443 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.0 2022-03-08T04:29:15.660 INFO:teuthology.orchestra.run.smithi018.stdout:94489280519 2022-03-08T04:29:15.660 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.9 2022-03-08T04:29:15.664 INFO:teuthology.orchestra.run.smithi018.stdout:90194313225 2022-03-08T04:29:15.664 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.5 2022-03-08T04:29:15.667 INFO:teuthology.orchestra.run.smithi018.stdout:107374182405 2022-03-08T04:29:15.668 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.11 2022-03-08T04:29:15.673 INFO:teuthology.orchestra.run.smithi018.stdout:85899345931 2022-03-08T04:29:15.674 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.1 2022-03-08T04:29:15.678 INFO:teuthology.orchestra.run.smithi018.stdout:111669149703 2022-03-08T04:29:15.678 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.3 2022-03-08T04:29:15.698 INFO:teuthology.orchestra.run.smithi018.stdout:120259084294 2022-03-08T04:29:15.699 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.4 2022-03-08T04:29:15.701 INFO:teuthology.orchestra.run.smithi018.stdout:81604378632 2022-03-08T04:29:15.701 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.8 2022-03-08T04:29:15.733 INFO:teuthology.orchestra.run.smithi018.stdout:120259084294 2022-03-08T04:29:15.734 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.7 2022-03-08T04:29:15.742 INFO:teuthology.orchestra.run.smithi018.stdout:115964116997 2022-03-08T04:29:15.742 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.6 2022-03-08T04:29:15.807 INFO:teuthology.orchestra.run.smithi018.stdout:98784247815 2022-03-08T04:29:15.807 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.10 2022-03-08T04:29:16.136 INFO:teuthology.orchestra.run.smithi018.stdout:90194313224 2022-03-08T04:29:16.144 INFO:teuthology.orchestra.run.smithi018.stdout:73014444040 2022-03-08T04:29:16.177 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.2 2022-03-08T04:29:16.499 INFO:teuthology.orchestra.run.smithi018.stdout:107374182404 2022-03-08T04:29:16.512 INFO:teuthology.orchestra.run.smithi018.stdout:94489280518 2022-03-08T04:29:16.602 INFO:teuthology.orchestra.run.smithi018.stdout:111669149702 2022-03-08T04:29:16.610 INFO:teuthology.orchestra.run.smithi018.stdout:120259084292 2022-03-08T04:29:16.651 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444041 got 73014444040 for osd.0 2022-03-08T04:29:16.656 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182405 got 107374182404 for osd.11 2022-03-08T04:29:16.661 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280519 got 94489280518 for osd.9 2022-03-08T04:29:16.665 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084294 got 120259084292 for osd.4 2022-03-08T04:29:16.670 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149703 got 111669149702 for osd.3 2022-03-08T04:29:16.755 INFO:teuthology.orchestra.run.smithi018.stdout:90194313224 2022-03-08T04:29:16.755 INFO:teuthology.orchestra.run.smithi018.stdout:81604378631 2022-03-08T04:29:16.780 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378632 got 81604378631 for osd.8 2022-03-08T04:29:16.784 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.5 2022-03-08T04:29:16.842 INFO:teuthology.orchestra.run.smithi018.stdout:115964116996 2022-03-08T04:29:16.861 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116996 for osd.6 2022-03-08T04:29:16.937 INFO:teuthology.orchestra.run.smithi018.stdout:120259084292 2022-03-08T04:29:16.956 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084294 got 120259084292 for osd.7 2022-03-08T04:29:16.975 INFO:teuthology.orchestra.run.smithi018.stdout:85899345930 2022-03-08T04:29:17.001 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345931 got 85899345930 for osd.1 2022-03-08T04:29:17.129 INFO:teuthology.orchestra.run.smithi018.stdout:98784247814 2022-03-08T04:29:17.148 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247815 got 98784247814 for osd.10 2022-03-08T04:29:17.178 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.2 2022-03-08T04:29:17.653 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.0 2022-03-08T04:29:17.656 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.11 2022-03-08T04:29:17.662 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.9 2022-03-08T04:29:17.666 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.4 2022-03-08T04:29:17.667 INFO:teuthology.orchestra.run.smithi018.stdout:90194313224 2022-03-08T04:29:17.670 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.3 2022-03-08T04:29:17.693 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.2 2022-03-08T04:29:17.781 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.8 2022-03-08T04:29:17.785 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.5 2022-03-08T04:29:17.862 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.6 2022-03-08T04:29:17.956 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.7 2022-03-08T04:29:18.002 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.1 2022-03-08T04:29:18.150 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.10 2022-03-08T04:29:18.445 INFO:teuthology.orchestra.run.smithi018.stdout:73014444041 2022-03-08T04:29:18.446 INFO:teuthology.orchestra.run.smithi018.stdout:107374182406 2022-03-08T04:29:18.472 INFO:teuthology.orchestra.run.smithi018.stdout:120259084294 2022-03-08T04:29:18.516 INFO:teuthology.orchestra.run.smithi018.stdout:94489280519 2022-03-08T04:29:18.577 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182405 got 107374182406 for osd.11 2022-03-08T04:29:18.578 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:18.580 INFO:teuthology.orchestra.run.smithi018.stdout:111669149703 2022-03-08T04:29:18.634 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444041 got 73014444041 for osd.0 2022-03-08T04:29:18.634 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:18.638 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084294 got 120259084294 for osd.4 2022-03-08T04:29:18.638 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:18.641 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280519 got 94489280519 for osd.9 2022-03-08T04:29:18.642 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:18.645 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149703 got 111669149703 for osd.3 2022-03-08T04:29:18.645 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:18.694 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph osd last-stat-seq osd.2 2022-03-08T04:29:18.713 INFO:teuthology.orchestra.run.smithi018.stdout:90194313225 2022-03-08T04:29:18.732 INFO:teuthology.orchestra.run.smithi018.stdout:81604378633 2022-03-08T04:29:18.738 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313225 got 90194313225 for osd.5 2022-03-08T04:29:18.739 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:18.760 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378632 got 81604378633 for osd.8 2022-03-08T04:29:18.760 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:18.904 INFO:teuthology.orchestra.run.smithi018.stdout:120259084294 2022-03-08T04:29:18.924 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084294 got 120259084294 for osd.7 2022-03-08T04:29:18.924 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:18.939 INFO:teuthology.orchestra.run.smithi018.stdout:115964116997 2022-03-08T04:29:18.963 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116997 for osd.6 2022-03-08T04:29:18.963 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:18.990 INFO:teuthology.orchestra.run.smithi018.stdout:85899345931 2022-03-08T04:29:19.009 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345931 got 85899345931 for osd.1 2022-03-08T04:29:19.010 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:19.011 INFO:teuthology.orchestra.run.smithi018.stdout:98784247815 2022-03-08T04:29:19.034 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247815 got 98784247815 for osd.10 2022-03-08T04:29:19.035 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:19.250 INFO:teuthology.orchestra.run.smithi018.stdout:90194313225 2022-03-08T04:29:19.275 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313225 got 90194313225 for osd.2 2022-03-08T04:29:19.275 DEBUG:teuthology.parallel:result is None 2022-03-08T04:29:19.276 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-08T04:29:19.276 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph pg dump --format=json 2022-03-08T04:29:19.810 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:19.813 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2022-03-08T04:29:19.835 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2022-03-08T04:29:19.007958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":65460,"kb_used_data":5012,"kb_used_omap":0,"kb_used_meta":59968,"kb_avail":1124876364,"statfs":{"total":1151940427776,"available":1151873396736,"internally_reserved":0,"allocated":5132288,"data_stored":3055893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61407232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":36,"num_write_kb":784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003862"},"pg_stats":[{"pgid":"1.0","version":"30'137","reported_seq":215,"reported_epoch":30,"state":"active+clean","last_fresh":"2022-03-08T04:29:13.198354+0000","last_change":"2022-03-08T04:29:02.008496+0000","last_active":"2022-03-08T04:29:13.198354+0000","last_peered":"2022-03-08T04:29:13.198354+0000","last_clean":"2022-03-08T04:29:13.198354+0000","last_became_active":"2022-03-08T04:29:01.999026+0000","last_became_peered":"2022-03-08T04:29:01.999026+0000","last_unstale":"2022-03-08T04:29:13.198354+0000","last_undegraded":"2022-03-08T04:29:13.198354+0000","last_fullsized":"2022-03-08T04:29:13.198354+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T04:28:51.743944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T04:28:51.743944+0000","last_clean_scrub_stamp":"2022-03-08T04:28:51.743944+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-09T09:26:31.811874+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1998848,"data_stored":1984624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":28,"seq":120259084294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6124,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739028,"statfs":{"total":95995035648,"available":95988764672,"internally_reserved":0,"allocated":823296,"data_stored":651065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7729999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1560000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.403}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.897}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4729999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]}]},{"osd":4,"up_from":28,"seq":120259084294,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5096,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740056,"statfs":{"total":95995035648,"available":95989817344,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]}]},{"osd":6,"up_from":27,"seq":115964116997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5096,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740056,"statfs":{"total":95995035648,"available":95989817344,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3410000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5019999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9570000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1579999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.399}]}]},{"osd":3,"up_from":26,"seq":111669149704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5096,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740056,"statfs":{"total":95995035648,"available":95989817344,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7569999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0019999999999998}]}]},{"osd":11,"up_from":25,"seq":107374182406,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9550000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.752}]}]},{"osd":10,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":827392,"data_stored":651380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8819999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.258}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4380000000000002}]}]},{"osd":8,"up_from":19,"seq":81604378633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6448,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738704,"statfs":{"total":95995035648,"available":95988432896,"internally_reserved":0,"allocated":827392,"data_stored":651380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.649}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1579999999999999}]}]},{"osd":1,"up_from":20,"seq":85899345931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.036}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1139999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0649999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]}]},{"osd":0,"up_from":17,"seq":73014444041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.637}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.369}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313226,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.744}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7360000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8360000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]}]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2450000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]}]},{"osd":5,"up_from":21,"seq":90194313226,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T04:29:19.838 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph pg dump --format=json 2022-03-08T04:29:20.382 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:20.385 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2022-03-08T04:29:20.401 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2022-03-08T04:29:19.007958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":65460,"kb_used_data":5012,"kb_used_omap":0,"kb_used_meta":59968,"kb_avail":1124876364,"statfs":{"total":1151940427776,"available":1151873396736,"internally_reserved":0,"allocated":5132288,"data_stored":3055893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61407232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":36,"num_write_kb":784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003862"},"pg_stats":[{"pgid":"1.0","version":"30'137","reported_seq":215,"reported_epoch":30,"state":"active+clean","last_fresh":"2022-03-08T04:29:13.198354+0000","last_change":"2022-03-08T04:29:02.008496+0000","last_active":"2022-03-08T04:29:13.198354+0000","last_peered":"2022-03-08T04:29:13.198354+0000","last_clean":"2022-03-08T04:29:13.198354+0000","last_became_active":"2022-03-08T04:29:01.999026+0000","last_became_peered":"2022-03-08T04:29:01.999026+0000","last_unstale":"2022-03-08T04:29:13.198354+0000","last_undegraded":"2022-03-08T04:29:13.198354+0000","last_fullsized":"2022-03-08T04:29:13.198354+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-08T04:28:51.743944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-08T04:28:51.743944+0000","last_clean_scrub_stamp":"2022-03-08T04:28:51.743944+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-09T09:26:31.811874+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1998848,"data_stored":1984624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":28,"seq":120259084294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6124,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739028,"statfs":{"total":95995035648,"available":95988764672,"internally_reserved":0,"allocated":823296,"data_stored":651065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7729999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1560000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.403}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.897}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4729999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]}]},{"osd":4,"up_from":28,"seq":120259084294,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5096,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740056,"statfs":{"total":95995035648,"available":95989817344,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]}]},{"osd":6,"up_from":27,"seq":115964116997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5096,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740056,"statfs":{"total":95995035648,"available":95989817344,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3410000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5019999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9570000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1579999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.399}]}]},{"osd":3,"up_from":26,"seq":111669149704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5096,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740056,"statfs":{"total":95995035648,"available":95989817344,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7569999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0019999999999998}]}]},{"osd":11,"up_from":25,"seq":107374182406,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9550000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.752}]}]},{"osd":10,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":827392,"data_stored":651380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8819999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.258}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4380000000000002}]}]},{"osd":8,"up_from":19,"seq":81604378633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6448,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738704,"statfs":{"total":95995035648,"available":95988432896,"internally_reserved":0,"allocated":827392,"data_stored":651380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.649}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1579999999999999}]}]},{"osd":1,"up_from":20,"seq":85899345931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.036}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1139999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0649999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]}]},{"osd":0,"up_from":17,"seq":73014444041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.637}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.369}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]}]},{"osd":2,"up_from":21,"seq":90194313226,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.744}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7360000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8360000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]}]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2450000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]}]},{"osd":5,"up_from":21,"seq":90194313226,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5160,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739992,"statfs":{"total":95995035648,"available":95989751808,"internally_reserved":0,"allocated":294912,"data_stored":122452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-08T04:29:20.403 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-08T04:29:20.403 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-08T04:29:20.403 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-08T04:29:20.404 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph health --format=json 2022-03-08T04:29:20.980 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:20.981 INFO:teuthology.orchestra.run.smithi018.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-08T04:29:21.007 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-08T04:29:21.007 INFO:tasks.rook:Rook complete, yielding 2022-03-08T04:29:21.008 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T04:29:21.019 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph -s 2022-03-08T04:29:21.617 INFO:teuthology.orchestra.run.smithi018.stdout: cluster: 2022-03-08T04:29:21.617 INFO:teuthology.orchestra.run.smithi018.stdout: id: 85af3f2f-7009-488d-b93d-f17d85a9eb4e 2022-03-08T04:29:21.618 INFO:teuthology.orchestra.run.smithi018.stdout: health: HEALTH_OK 2022-03-08T04:29:21.618 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:21.618 INFO:teuthology.orchestra.run.smithi018.stdout: services: 2022-03-08T04:29:21.618 INFO:teuthology.orchestra.run.smithi018.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-08T04:29:21.619 INFO:teuthology.orchestra.run.smithi018.stdout: mgr: a(active, since 2m) 2022-03-08T04:29:21.619 INFO:teuthology.orchestra.run.smithi018.stdout: osd: 12 osds: 12 up (since 22s), 12 in (since 76s) 2022-03-08T04:29:21.619 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:21.620 INFO:teuthology.orchestra.run.smithi018.stdout: data: 2022-03-08T04:29:21.620 INFO:teuthology.orchestra.run.smithi018.stdout: pools: 1 pools, 1 pgs 2022-03-08T04:29:21.620 INFO:teuthology.orchestra.run.smithi018.stdout: objects: 2 objects, 577 KiB 2022-03-08T04:29:21.621 INFO:teuthology.orchestra.run.smithi018.stdout: usage: 64 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-08T04:29:21.621 INFO:teuthology.orchestra.run.smithi018.stdout: pgs: 1 active+clean 2022-03-08T04:29:21.621 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:21.621 INFO:teuthology.orchestra.run.smithi018.stdout: io: 2022-03-08T04:29:21.622 INFO:teuthology.orchestra.run.smithi018.stdout: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr 2022-03-08T04:29:21.622 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:29:21.650 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch status 2022-03-08T04:29:22.375 INFO:teuthology.orchestra.run.smithi018.stdout:Backend: rook 2022-03-08T04:29:22.375 INFO:teuthology.orchestra.run.smithi018.stdout:Available: Yes 2022-03-08T04:29:22.400 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ps 2022-03-08T04:29:23.374 INFO:teuthology.orchestra.run.smithi018.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T04:29:23.374 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi018 running (77s) 0s ago 78s - - 4b871cde5e1a 2022-03-08T04:29:23.374 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi084 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-08T04:29:23.375 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi105 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-08T04:29:23.375 INFO:teuthology.orchestra.run.smithi018.stdout:mgr.a smithi018 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-08T04:29:23.375 INFO:teuthology.orchestra.run.smithi018.stdout:mon.a smithi105 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T04:29:23.375 INFO:teuthology.orchestra.run.smithi018.stdout:mon.b smithi084 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T04:29:23.376 INFO:teuthology.orchestra.run.smithi018.stdout:mon.c smithi018 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:29:23.376 INFO:teuthology.orchestra.run.smithi018.stdout:osd.0 smithi084 running (81s) 0s ago 81s - - 4b871cde5e1a 2022-03-08T04:29:23.376 INFO:teuthology.orchestra.run.smithi018.stdout:osd.1 smithi084 running (76s) 0s ago 77s - - 4b871cde5e1a 2022-03-08T04:29:23.376 INFO:teuthology.orchestra.run.smithi018.stdout:osd.10 smithi105 running (64s) 0s ago 65s - - 4b871cde5e1a 2022-03-08T04:29:23.376 INFO:teuthology.orchestra.run.smithi018.stdout:osd.11 smithi105 running (62s) 0s ago 63s - - 4b871cde5e1a 2022-03-08T04:29:23.377 INFO:teuthology.orchestra.run.smithi018.stdout:osd.2 smithi084 running (73s) 0s ago 74s - - 4b871cde5e1a 2022-03-08T04:29:23.377 INFO:teuthology.orchestra.run.smithi018.stdout:osd.3 smithi018 running (78s) 0s ago 79s - - 4b871cde5e1a 2022-03-08T04:29:23.377 INFO:teuthology.orchestra.run.smithi018.stdout:osd.4 smithi018 running (73s) 0s ago 73s - - 4b871cde5e1a 2022-03-08T04:29:23.377 INFO:teuthology.orchestra.run.smithi018.stdout:osd.5 smithi084 running (67s) 0s ago 67s - - 4b871cde5e1a 2022-03-08T04:29:23.378 INFO:teuthology.orchestra.run.smithi018.stdout:osd.6 smithi018 running (69s) 0s ago 69s - - 4b871cde5e1a 2022-03-08T04:29:23.378 INFO:teuthology.orchestra.run.smithi018.stdout:osd.7 smithi018 running (70s) 0s ago 70s - - 4b871cde5e1a 2022-03-08T04:29:23.378 INFO:teuthology.orchestra.run.smithi018.stdout:osd.8 smithi105 running (71s) 0s ago 72s - - 4b871cde5e1a 2022-03-08T04:29:23.378 INFO:teuthology.orchestra.run.smithi018.stdout:osd.9 smithi105 running (65s) 0s ago 66s - - 4b871cde5e1a 2022-03-08T04:29:23.396 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls 2022-03-08T04:29:24.162 INFO:teuthology.orchestra.run.smithi018.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T04:29:24.163 INFO:teuthology.orchestra.run.smithi018.stdout:crash 3/3 0s ago 2m * 2022-03-08T04:29:24.163 INFO:teuthology.orchestra.run.smithi018.stdout:mgr 1/1 0s ago 2m count:1 2022-03-08T04:29:24.163 INFO:teuthology.orchestra.run.smithi018.stdout:mon 3/3 0s ago 5m count:3 2022-03-08T04:29:24.164 INFO:teuthology.orchestra.run.smithi018.stdout:osd 24 0s ago 82s 2022-03-08T04:29:24.164 INFO:teuthology.orchestra.run.smithi018.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T04:29:24.200 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch device ls 2022-03-08T04:29:24.750 INFO:teuthology.orchestra.run.smithi018.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T04:29:24.750 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T04:29:24.751 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T04:29:24.751 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T04:29:24.751 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T04:29:24.751 INFO:teuthology.orchestra.run.smithi018.stdout:smithi084 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T04:29:24.752 INFO:teuthology.orchestra.run.smithi018.stdout:smithi084 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T04:29:24.752 INFO:teuthology.orchestra.run.smithi018.stdout:smithi084 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T04:29:24.752 INFO:teuthology.orchestra.run.smithi018.stdout:smithi084 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T04:29:24.752 INFO:teuthology.orchestra.run.smithi018.stdout:smithi105 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T04:29:24.753 INFO:teuthology.orchestra.run.smithi018.stdout:smithi105 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T04:29:24.753 INFO:teuthology.orchestra.run.smithi018.stdout:smithi105 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T04:29:24.753 INFO:teuthology.orchestra.run.smithi018.stdout:smithi105 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T04:29:24.769 INFO:teuthology.run_tasks:Running task exec... 2022-03-08T04:29:24.782 INFO:teuthology.task.exec:Executing custom commands... 2022-03-08T04:29:24.782 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi018.front.sepia.ceph.com 2022-03-08T04:29:24.783 DEBUG:teuthology.orchestra.run.smithi018:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-08T04:29:24.783 DEBUG:teuthology.orchestra.run.smithi018:> toolbox() { 2022-03-08T04:29:24.783 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-08T04:29:24.784 DEBUG:teuthology.orchestra.run.smithi018:> } 2022-03-08T04:29:24.784 DEBUG:teuthology.orchestra.run.smithi018:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T04:29:24.784 DEBUG:teuthology.orchestra.run.smithi018:> toolbox ceph orch osd rm 0 --force 2022-03-08T04:29:24.784 DEBUG:teuthology.orchestra.run.smithi018:> removed_pv="" 2022-03-08T04:29:24.785 DEBUG:teuthology.orchestra.run.smithi018:> while [ "$removed_pv" = "" ] 2022-03-08T04:29:24.785 DEBUG:teuthology.orchestra.run.smithi018:> do 2022-03-08T04:29:24.785 DEBUG:teuthology.orchestra.run.smithi018:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-08T04:29:24.786 DEBUG:teuthology.orchestra.run.smithi018:> sleep 3s 2022-03-08T04:29:24.786 DEBUG:teuthology.orchestra.run.smithi018:> done 2022-03-08T04:29:24.786 DEBUG:teuthology.orchestra.run.smithi018:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-08T04:29:24.786 DEBUG:teuthology.orchestra.run.smithi018:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-08T04:29:24.788 DEBUG:teuthology.orchestra.run.smithi018:> toolbox ceph orch device zap $host $target_path --force 2022-03-08T04:29:24.788 DEBUG:teuthology.orchestra.run.smithi018:> zap_completion="0" 2022-03-08T04:29:24.789 DEBUG:teuthology.orchestra.run.smithi018:> while [ "$zap_completion" = "0" ] 2022-03-08T04:29:24.789 DEBUG:teuthology.orchestra.run.smithi018:> do 2022-03-08T04:29:24.790 DEBUG:teuthology.orchestra.run.smithi018:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-08T04:29:24.790 DEBUG:teuthology.orchestra.run.smithi018:> sleep 3s 2022-03-08T04:29:24.790 DEBUG:teuthology.orchestra.run.smithi018:> done 2022-03-08T04:29:24.791 DEBUG:teuthology.orchestra.run.smithi018:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-08T04:29:24.791 DEBUG:teuthology.orchestra.run.smithi018:> toolbox ceph orch apply osd --all-available-devices 2022-03-08T04:29:24.791 DEBUG:teuthology.orchestra.run.smithi018:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T04:29:24.791 DEBUG:teuthology.orchestra.run.smithi018:> num_osd="0" 2022-03-08T04:29:24.792 DEBUG:teuthology.orchestra.run.smithi018:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-08T04:29:24.792 DEBUG:teuthology.orchestra.run.smithi018:> do 2022-03-08T04:29:24.792 DEBUG:teuthology.orchestra.run.smithi018:> echo "waiting for osd to come back up" 2022-03-08T04:29:24.793 DEBUG:teuthology.orchestra.run.smithi018:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-08T04:29:24.793 DEBUG:teuthology.orchestra.run.smithi018:> sleep 30s 2022-03-08T04:29:24.793 DEBUG:teuthology.orchestra.run.smithi018:> done 2022-03-08T04:29:24.793 DEBUG:teuthology.orchestra.run.smithi018:> ' 2022-03-08T04:29:24.811 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-08T04:29:24.811 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T04:29:24.811 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-08T04:29:27.480 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T04:29:27.962 INFO:teuthology.orchestra.run.smithi018.stderr:+ orig_num_osd=12 2022-03-08T04:29:27.962 INFO:teuthology.orchestra.run.smithi018.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-08T04:29:27.962 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-08T04:29:28.044 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T04:29:28.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:28.661044 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T04:29:28.663 INFO:tasks.rook.operator.smithi018.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T04:29:28.663 INFO:tasks.rook.operator.smithi018.stdout: Storage: v1.StorageScopeSpec{ 2022-03-08T04:29:28.664 INFO:tasks.rook.operator.smithi018.stdout: ... // 3 identical fields 2022-03-08T04:29:28.664 INFO:tasks.rook.operator.smithi018.stdout: Config: nil, 2022-03-08T04:29:28.664 INFO:tasks.rook.operator.smithi018.stdout: Selection: {}, 2022-03-08T04:29:28.664 INFO:tasks.rook.operator.smithi018.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T04:29:28.665 INFO:tasks.rook.operator.smithi018.stdout: ... // 2 identical elements 2022-03-08T04:29:28.665 INFO:tasks.rook.operator.smithi018.stdout: {Name: "smithi018-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi018-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T04:29:28.665 INFO:tasks.rook.operator.smithi018.stdout: {Name: "smithi018-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi018-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T04:29:28.665 INFO:tasks.rook.operator.smithi018.stdout:- { 2022-03-08T04:29:28.665 INFO:tasks.rook.operator.smithi018.stdout:- Name: "smithi084-lv-1", 2022-03-08T04:29:28.666 INFO:tasks.rook.operator.smithi018.stdout:- Count: 1, 2022-03-08T04:29:28.666 INFO:tasks.rook.operator.smithi018.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T04:29:28.666 INFO:tasks.rook.operator.smithi018.stdout:- }, 2022-03-08T04:29:28.666 INFO:tasks.rook.operator.smithi018.stdout: {Name: "smithi084-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi084-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-08T04:29:28.667 INFO:tasks.rook.operator.smithi018.stdout: {Name: "smithi084-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi084-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T04:29:28.667 INFO:tasks.rook.operator.smithi018.stdout: ... // 5 identical elements 2022-03-08T04:29:28.667 INFO:tasks.rook.operator.smithi018.stdout: }, 2022-03-08T04:29:28.667 INFO:tasks.rook.operator.smithi018.stdout: }, 2022-03-08T04:29:28.668 INFO:tasks.rook.operator.smithi018.stdout: Annotations: nil, 2022-03-08T04:29:28.668 INFO:tasks.rook.operator.smithi018.stdout: Labels: nil, 2022-03-08T04:29:28.668 INFO:tasks.rook.operator.smithi018.stdout: ... // 20 identical fields 2022-03-08T04:29:28.668 INFO:tasks.rook.operator.smithi018.stdout: } 2022-03-08T04:29:28.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:28.661164 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T04:29:28.669 INFO:tasks.rook.operator.smithi018.stdout:I0308 04:29:28.661447 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T04:29:28.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:28.661582 I | operator: successfully started the controller-runtime manager 2022-03-08T04:29:28.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:28.681797 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T04:29:28.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:28.681826 I | operator: watching all namespaces for Ceph CRs 2022-03-08T04:29:28.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:28.681868 I | operator: setting up schemes 2022-03-08T04:29:28.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:28.683980 I | operator: setting up the controller-runtime manager 2022-03-08T04:29:28.988 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:28.988108 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T04:29:29.089 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.088617 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T04:29:29.089 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.088696 I | ceph-cluster-controller: successfully started 2022-03-08T04:29:29.089 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.088759 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T04:29:29.090 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.088782 I | ceph-crashcollector-controller: successfully started 2022-03-08T04:29:29.090 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.088809 I | ceph-block-pool-controller: successfully started 2022-03-08T04:29:29.091 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.088835 I | ceph-object-store-user-controller: successfully started 2022-03-08T04:29:29.091 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.088857 I | ceph-object-realm-controller: successfully started 2022-03-08T04:29:29.091 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.088868 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T04:29:29.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.088880 I | ceph-object-zone-controller: successfully started 2022-03-08T04:29:29.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089056 I | ceph-object-controller: successfully started 2022-03-08T04:29:29.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089116 I | ceph-file-controller: successfully started 2022-03-08T04:29:29.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089153 I | ceph-nfs-controller: successfully started 2022-03-08T04:29:29.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089196 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T04:29:29.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089235 I | ceph-client-controller: successfully started 2022-03-08T04:29:29.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089273 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T04:29:29.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089321 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T04:29:29.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089350 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T04:29:29.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089374 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T04:29:29.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089406 I | ceph-bucket-topic: successfully started 2022-03-08T04:29:29.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089428 I | ceph-bucket-notification: successfully started 2022-03-08T04:29:29.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089440 I | ceph-bucket-notification: successfully started 2022-03-08T04:29:29.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.089462 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T04:29:29.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.090922 I | operator: starting the controller-runtime manager 2022-03-08T04:29:29.195 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.194185 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T04:29:29.200 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.200159 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T04:29:29.200 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.200203 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T04:29:29.200 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.200216 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T04:29:29.212 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.212619 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T04:29:29.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.214811 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:29:29.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.214846 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:29:29.216 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.214934 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T04:29:29.216 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.214950 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T04:29:29.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.216166 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T04:29:29.218 INFO:tasks.rook.operator.smithi018.stdout:I0308 04:29:29.216226 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T04:29:29.229 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.229437 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T04:29:29.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.236316 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T04:29:29.255 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254079 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T04:29:29.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254103 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T04:29:29.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254110 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T04:29:29.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254117 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T04:29:29.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254125 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T04:29:29.257 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254131 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T04:29:29.257 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254138 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T04:29:29.257 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254144 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T04:29:29.257 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254151 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T04:29:29.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254157 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T04:29:29.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254162 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T04:29:29.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254169 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T04:29:29.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254175 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T04:29:29.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254180 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T04:29:29.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254186 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T04:29:29.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254193 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T04:29:29.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254283 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:29:29.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:29.254301 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:29:31.893 INFO:teuthology.orchestra.run.smithi018.stdout:Success 2022-03-08T04:29:31.913 INFO:teuthology.orchestra.run.smithi018.stderr:+ removed_pv= 2022-03-08T04:29:31.914 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' '' = '' ']' 2022-03-08T04:29:31.915 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl get pv 2022-03-08T04:29:31.915 INFO:teuthology.orchestra.run.smithi018.stderr:++ grep Released 2022-03-08T04:29:31.916 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f1 -d ' ' 2022-03-08T04:29:32.051 INFO:teuthology.orchestra.run.smithi018.stderr:+ removed_pv= 2022-03-08T04:29:32.051 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 3s 2022-03-08T04:29:34.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:34.077626 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-08T04:29:35.053 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' '' = '' ']' 2022-03-08T04:29:35.053 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl get pv 2022-03-08T04:29:35.054 INFO:teuthology.orchestra.run.smithi018.stderr:++ grep Released 2022-03-08T04:29:35.054 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f1 -d ' ' 2022-03-08T04:29:35.298 INFO:teuthology.orchestra.run.smithi018.stderr:+ removed_pv=smithi084-lv-1 2022-03-08T04:29:35.298 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 3s 2022-03-08T04:29:37.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079487 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T04:29:37.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079527 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T04:29:37.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079538 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T04:29:37.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079545 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T04:29:37.082 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079552 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T04:29:37.082 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079558 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T04:29:37.082 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079565 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T04:29:37.082 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079571 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T04:29:37.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079577 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T04:29:37.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079584 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T04:29:37.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079590 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T04:29:37.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079596 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T04:29:37.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079602 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T04:29:37.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079609 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T04:29:37.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079615 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T04:29:37.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079623 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T04:29:37.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079629 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T04:29:37.085 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079635 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T04:29:37.085 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079641 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T04:29:37.085 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079648 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T04:29:37.085 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079655 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T04:29:37.085 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079662 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T04:29:37.086 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079667 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T04:29:37.086 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079675 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T04:29:37.086 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.079681 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T04:29:37.443 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.443971 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T04:29:37.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.455169 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:29:37.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.455197 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:29:37.457 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.455203 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:29:37.457 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.455206 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:29:37.457 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.455210 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:29:37.457 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.455214 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:29:37.458 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.455221 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T04:29:37.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.686431 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:29:37.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.686452 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:29:37.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.686455 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T04:29:37.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:37.946738 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T04:29:38.299 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' smithi084-lv-1 = '' ']' 2022-03-08T04:29:38.300 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl get pv smithi084-lv-1 -o 'jsonpath={.spec.local.path}' 2022-03-08T04:29:38.561 INFO:teuthology.orchestra.run.smithi018.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-03-08T04:29:38.562 INFO:teuthology.orchestra.run.smithi018.stderr:++ echo smithi084-lv-1 2022-03-08T04:29:38.563 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f1 -d - 2022-03-08T04:29:38.564 INFO:teuthology.orchestra.run.smithi018.stderr:+ host=smithi084 2022-03-08T04:29:38.565 INFO:teuthology.orchestra.run.smithi018.stderr:+ toolbox ceph orch device zap smithi084 /dev/vg_nvme/lv_1 --force 2022-03-08T04:29:38.565 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi084 /dev/vg_nvme/lv_1 --force 2022-03-08T04:29:38.853 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:38.853651 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:29:38.854 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:38.853685 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:29:38.854 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:38.853694 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T04:29:38.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:38.857371 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T04:29:38.859 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:38.857404 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T04:29:38.859 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:38.857435 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T04:29:38.890 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T04:29:38.993 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:38.993244 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:29:39.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.019205 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:29:39.028 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.019500 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:29:39.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.019614 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:29:39.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.019646 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:29:39.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.019659 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T04:29:39.068 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.068112 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T04:29:39.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.325896 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T04:29:39.331 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.329737 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T04:29:39.331 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.329752 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T04:29:39.349 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.349501 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T04:29:39.454 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.454421 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T04:29:39.517 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_1 on smithi084 zapped 2022-03-08T04:29:39.538 INFO:teuthology.orchestra.run.smithi018.stderr:+ zap_completion=0 2022-03-08T04:29:39.538 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 0 = 0 ']' 2022-03-08T04:29:39.538 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-08T04:29:39.676 INFO:teuthology.orchestra.run.smithi018.stderr:+ zap_completion= 2022-03-08T04:29:39.676 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 3s 2022-03-08T04:29:39.680 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.680734 I | op-mon: start running mons 2022-03-08T04:29:39.712 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.712696 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:29:39.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:39.830136 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.14.128:6789","10.104.177.194:6789","10.99.92.113:6789"]}] data:c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:2] 2022-03-08T04:29:40.527 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:40.527118 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:29:40.527 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:40.527401 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:29:41.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:41.967378 I | op-mon: targeting the mon count 3 2022-03-08T04:29:41.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:41.976326 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:29:42.327 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:42.326904 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:29:42.327 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:42.326933 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:29:42.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:42.653370 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:29:42.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:42.653420 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:29:42.677 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' '' = 0 ']' 2022-03-08T04:29:42.677 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl patch pv smithi084-lv-1 -p '{"spec":{"claimRef": null}}' 2022-03-08T04:29:42.746 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi084-lv-1 patched 2022-03-08T04:29:42.748 INFO:teuthology.orchestra.run.smithi018.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-08T04:29:42.748 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-08T04:29:42.819 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T04:29:42.998 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:42.998800 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:29:42.999 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:42.998830 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:29:43.426 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.426135 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:29:43.426 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.426163 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T04:29:43.494 INFO:teuthology.orchestra.run.smithi018.stdout:['Success'] 2022-03-08T04:29:43.503 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.499907 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-08T04:29:43.504 INFO:tasks.rook.operator.smithi018.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-08T04:29:43.504 INFO:tasks.rook.operator.smithi018.stdout: Storage: v1.StorageScopeSpec{ 2022-03-08T04:29:43.505 INFO:tasks.rook.operator.smithi018.stdout: ... // 3 identical fields 2022-03-08T04:29:43.505 INFO:tasks.rook.operator.smithi018.stdout: Config: nil, 2022-03-08T04:29:43.506 INFO:tasks.rook.operator.smithi018.stdout: Selection: {}, 2022-03-08T04:29:43.506 INFO:tasks.rook.operator.smithi018.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-08T04:29:43.506 INFO:tasks.rook.operator.smithi018.stdout: ... // 9 identical elements 2022-03-08T04:29:43.507 INFO:tasks.rook.operator.smithi018.stdout: {Name: "smithi105-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi105-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-08T04:29:43.507 INFO:tasks.rook.operator.smithi018.stdout: {Name: "smithi105-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi105-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-08T04:29:43.508 INFO:tasks.rook.operator.smithi018.stdout:+ { 2022-03-08T04:29:43.508 INFO:tasks.rook.operator.smithi018.stdout:+ Name: "smithi084-lv-1", 2022-03-08T04:29:43.508 INFO:tasks.rook.operator.smithi018.stdout:+ Count: 1, 2022-03-08T04:29:43.509 INFO:tasks.rook.operator.smithi018.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-08T04:29:43.509 INFO:tasks.rook.operator.smithi018.stdout:+ }, 2022-03-08T04:29:43.510 INFO:tasks.rook.operator.smithi018.stdout: }, 2022-03-08T04:29:43.510 INFO:tasks.rook.operator.smithi018.stdout: }, 2022-03-08T04:29:43.510 INFO:tasks.rook.operator.smithi018.stdout: Annotations: nil, 2022-03-08T04:29:43.511 INFO:tasks.rook.operator.smithi018.stdout: Labels: nil, 2022-03-08T04:29:43.511 INFO:tasks.rook.operator.smithi018.stdout: ... // 20 identical fields 2022-03-08T04:29:43.511 INFO:tasks.rook.operator.smithi018.stdout: } 2022-03-08T04:29:43.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.499994 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T04:29:43.512 INFO:tasks.rook.operator.smithi018.stdout:I0308 04:29:43.500238 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T04:29:43.517 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-08T04:29:43.585 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.585468 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T04:29:43.586 INFO:teuthology.orchestra.run.smithi018.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-08T04:29:43.587 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.585491 I | operator: watching all namespaces for Ceph CRs 2022-03-08T04:29:43.587 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.585521 I | operator: setting up schemes 2022-03-08T04:29:43.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.587956 I | operator: setting up the controller-runtime manager 2022-03-08T04:29:43.593 INFO:teuthology.orchestra.run.smithi018.stdout:waiting for osd to come back up 2022-03-08T04:29:43.594 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=0 2022-03-08T04:29:43.594 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 0 '!=' 12 ']' 2022-03-08T04:29:43.594 INFO:teuthology.orchestra.run.smithi018.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T04:29:43.594 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-08T04:29:43.595 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T04:29:43.595 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-08T04:29:43.766 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T04:29:43.891 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.891815 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T04:29:43.896 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.894714 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T04:29:43.896 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.894744 I | ceph-cluster-controller: successfully started 2022-03-08T04:29:43.897 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.894800 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T04:29:43.898 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.894825 I | ceph-crashcollector-controller: successfully started 2022-03-08T04:29:43.898 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.894861 I | ceph-block-pool-controller: successfully started 2022-03-08T04:29:43.898 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.894881 I | ceph-object-store-user-controller: successfully started 2022-03-08T04:29:43.898 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.894905 I | ceph-object-realm-controller: successfully started 2022-03-08T04:29:43.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.894920 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T04:29:43.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.894932 I | ceph-object-zone-controller: successfully started 2022-03-08T04:29:43.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895065 I | ceph-object-controller: successfully started 2022-03-08T04:29:43.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895102 I | ceph-file-controller: successfully started 2022-03-08T04:29:43.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895149 I | ceph-nfs-controller: successfully started 2022-03-08T04:29:43.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895185 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T04:29:43.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895219 I | ceph-client-controller: successfully started 2022-03-08T04:29:43.901 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895244 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T04:29:43.901 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895290 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T04:29:43.901 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895314 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T04:29:43.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895339 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T04:29:43.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895363 I | ceph-bucket-topic: successfully started 2022-03-08T04:29:43.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895381 I | ceph-bucket-notification: successfully started 2022-03-08T04:29:43.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895402 I | ceph-bucket-notification: successfully started 2022-03-08T04:29:43.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.895422 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T04:29:43.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.896433 I | operator: starting the controller-runtime manager 2022-03-08T04:29:43.928 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.928796 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T04:29:43.929 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.928817 I | op-mon: checking for basic quorum with existing mons 2022-03-08T04:29:43.972 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.972036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-03-08T04:29:43.973 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.972165 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T04:29:43.992 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.992892 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:29:43.997 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.997658 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T04:29:43.997 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.997686 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T04:29:43.997 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:43.997695 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T04:29:43.999 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.000396 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T04:29:44.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.002513 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T04:29:44.006 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.004099 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:29:44.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.004143 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-08T04:29:44.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.004151 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-08T04:29:44.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.005447 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-08T04:29:44.008 INFO:tasks.rook.operator.smithi018.stdout:I0308 04:29:44.005489 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-08T04:29:44.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.006328 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:29:44.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.042890 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T04:29:44.046 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.044198 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-03-08 04:29:44.014171528 +0000 UTC m=+481.158238603 LastTransitionTime:2022-03-08 04:29:44.01417145 +0000 UTC m=+481.158238532}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:29:44.046 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.044211 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T04:29:44.062 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.063019 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-08T04:29:44.197 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=11 2022-03-08T04:29:44.198 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 30s 2022-03-08T04:29:44.493 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.493124 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-08T04:29:44.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.552174 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T04:29:44.628 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627191 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-08T04:29:44.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627239 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-08T04:29:44.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627252 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-08T04:29:44.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627262 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-08T04:29:44.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627275 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-08T04:29:44.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627287 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-08T04:29:44.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627301 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-08T04:29:44.631 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627317 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-08T04:29:44.631 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627332 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-08T04:29:44.631 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627346 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-08T04:29:44.631 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627359 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-08T04:29:44.632 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627376 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-08T04:29:44.632 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627390 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-08T04:29:44.632 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627401 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-08T04:29:44.632 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627414 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-08T04:29:44.632 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627430 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-08T04:29:44.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627527 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:29:44.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:44.627553 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:29:47.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:47.557341 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T04:29:50.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:50.560236 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-08T04:29:52.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375392 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-08T04:29:52.466 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375434 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-08T04:29:52.466 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375446 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T04:29:52.466 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375453 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-08T04:29:52.466 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375461 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T04:29:52.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375467 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-08T04:29:52.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375476 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T04:29:52.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375484 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-08T04:29:52.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375492 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T04:29:52.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375502 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-08T04:29:52.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375510 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T04:29:52.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375522 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-08T04:29:52.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375533 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-08T04:29:52.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375548 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-08T04:29:52.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375559 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-08T04:29:52.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375567 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-08T04:29:52.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375578 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-08T04:29:52.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375588 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-08T04:29:52.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375596 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-08T04:29:52.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375604 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T04:29:52.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375621 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-08T04:29:52.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375635 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-08T04:29:52.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375645 I | ceph-csi: Kubernetes version is 1.23 2022-03-08T04:29:52.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375655 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-08T04:29:52.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.375665 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-08T04:29:52.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.384430 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-08T04:29:52.472 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.393152 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:29:52.472 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.393168 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:29:52.472 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.393172 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:29:52.472 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.393175 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:29:52.473 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.393178 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:29:52.473 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.393181 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:29:52.473 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.393185 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-08T04:29:52.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.543019 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:29:52.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.543050 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:29:52.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.543074 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-08T04:29:52.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.588004 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-08T04:29:52.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.803733 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-08T04:29:52.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.803755 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-08T04:29:52.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.803761 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-08T04:29:52.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.851205 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-08T04:29:52.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.851236 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-08T04:29:52.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.851244 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-08T04:29:52.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:52.892193 I | ceph-csi: successfully started CSI CephFS driver 2022-03-08T04:29:53.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:53.151229 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T04:29:53.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:53.157443 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-08T04:29:53.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:53.157469 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-08T04:29:53.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:53.162232 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-08T04:29:53.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:53.564746 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-08T04:29:59.619 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.618705 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T04:29:59.620 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.618731 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T04:29:59.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.618744 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T04:29:59.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.627838 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T04:29:59.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.627864 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-08T04:29:59.628 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.627877 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-08T04:29:59.631 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.629140 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:29:59.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.710571 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:29:59.714 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.710855 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:29:59.726 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.727263 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:29:59.734 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.734890 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:29:59.734 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:29:59.735098 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:30:00.140 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:00.141037 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T04:30:00.238 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:00.238643 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-08T04:30:00.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:00.250478 I | op-mon: start running mons 2022-03-08T04:30:00.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:00.277017 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:30:01.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:01.344598 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.92.113:6789","10.110.14.128:6789","10.104.177.194:6789"]}] data:c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:2] 2022-03-08T04:30:01.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:01.943426 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:30:01.955 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:01.943765 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:30:03.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:03.545375 I | op-mon: targeting the mon count 3 2022-03-08T04:30:03.556 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:03.556956 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:30:03.913 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:03.913338 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:30:03.913 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:03.913367 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:30:04.252 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:04.252522 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:30:04.253 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:04.252553 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:30:04.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:04.660867 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:30:04.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:04.660902 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:30:04.993 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:04.992652 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:30:04.993 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:04.992716 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T04:30:05.330 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:05.330349 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T04:30:05.331 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:05.330380 I | op-mon: checking for basic quorum with existing mons 2022-03-08T04:30:05.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:05.467049 I | op-mon: mon "b" endpoint is [v2:10.99.92.113:3300,v1:10.99.92.113:6789] 2022-03-08T04:30:05.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:05.595655 I | op-mon: mon "c" endpoint is [v2:10.110.14.128:3300,v1:10.110.14.128:6789] 2022-03-08T04:30:05.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:05.943711 I | op-mon: mon "a" endpoint is [v2:10.104.177.194:3300,v1:10.104.177.194:6789] 2022-03-08T04:30:06.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:06.545234 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.14.128:6789","10.104.177.194:6789","10.99.92.113:6789"]}] data:c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:2] 2022-03-08T04:30:07.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:07.146521 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:30:07.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:07.146832 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:30:07.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:07.564408 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T04:30:07.580 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:07.580481 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T04:30:07.581 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:07.580514 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T04:30:08.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:08.151732 I | op-mon: mons running: [b c a] 2022-03-08T04:30:08.577 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:08.577703 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:30:08.586 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:08.586567 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T04:30:08.599 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:08.599070 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T04:30:08.599 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:08.599104 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T04:30:08.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:08.754668 I | op-mon: mons running: [b c a] 2022-03-08T04:30:09.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:09.172722 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:30:09.180 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:09.180537 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T04:30:09.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:09.187382 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T04:30:09.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:09.187399 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T04:30:09.353 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:09.353775 I | op-mon: mons running: [b c a] 2022-03-08T04:30:09.826 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:09.826502 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:30:09.827 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:09.826532 I | op-mon: mons created: 3 2022-03-08T04:30:10.262 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:10.262462 I | op-mon: waiting for mon quorum with [b c a] 2022-03-08T04:30:10.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:10.303575 I | op-mon: mons running: [b c a] 2022-03-08T04:30:10.727 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:10.727316 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:30:10.727 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:10.727364 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T04:30:10.728 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:10.727432 I | op-mon: start running mons 2022-03-08T04:30:10.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:10.737737 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:30:11.181 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:11.181804 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T04:30:11.343 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:11.343918 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.14.128:6789","10.104.177.194:6789","10.99.92.113:6789"]}] data:c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:2] 2022-03-08T04:30:11.628 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:11.628405 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T04:30:11.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:11.943223 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:30:11.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:11.943573 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:30:12.207 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:12.207124 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T04:30:14.199 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T04:30:14.200 INFO:teuthology.orchestra.run.smithi018.stdout:waiting for osd to come back up 2022-03-08T04:30:14.201 INFO:teuthology.orchestra.run.smithi018.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T04:30:14.201 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-08T04:30:14.201 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T04:30:14.202 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-08T04:30:14.272 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T04:30:14.343 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:14.343514 I | op-mon: targeting the mon count 3 2022-03-08T04:30:14.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:14.352085 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:30:14.763 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:14.763549 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-08T04:30:14.763 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:14.763591 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:30:14.767 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=11 2022-03-08T04:30:14.767 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 30s 2022-03-08T04:30:15.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:15.115087 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-08T04:30:15.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:15.115138 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:30:15.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:15.145000 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T04:30:15.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:15.145024 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T04:30:15.450 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:15.450741 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-08T04:30:15.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:15.450769 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:30:15.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:15.568738 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T04:30:15.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:15.900246 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-08T04:30:15.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:15.900277 I | op-config: deleting "log file" option from the mon configuration database 2022-03-08T04:30:16.000 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:16.000054 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T04:30:16.000 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:16.000093 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T04:30:16.000 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:16.000104 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T04:30:16.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:16.233177 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-08T04:30:16.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:16.233219 I | op-mon: checking for basic quorum with existing mons 2022-03-08T04:30:16.343 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:16.343248 I | op-mon: mon "c" endpoint is [v2:10.110.14.128:3300,v1:10.110.14.128:6789] 2022-03-08T04:30:16.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:16.406224 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T04:30:17.218 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:17.218856 I | op-mgr: start running mgr 2022-03-08T04:30:17.219 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:17.218899 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T04:30:17.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:17.545523 I | op-mon: mon "a" endpoint is [v2:10.104.177.194:3300,v1:10.104.177.194:6789] 2022-03-08T04:30:18.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:18.774217 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T04:30:18.782 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:18.782485 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T04:30:18.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:18.785733 I | op-mgr: setting services to point to mgr "a" 2022-03-08T04:30:18.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:18.943780 I | op-mon: mon "b" endpoint is [v2:10.99.92.113:3300,v1:10.99.92.113:6789] 2022-03-08T04:30:20.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:20.145430 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.110.14.128:6789","10.104.177.194:6789","10.99.92.113:6789"]}] data:c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 mapping:{"node":{"a":{"Name":"smithi105","Hostname":"smithi105","Address":"172.21.15.105"},"b":{"Name":"smithi084","Hostname":"smithi084","Address":"172.21.15.84"},"c":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"}}} maxMonId:2] 2022-03-08T04:30:20.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:20.743331 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T04:30:20.744 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:20.743439 I | op-mgr: successful modules: balancer 2022-03-08T04:30:20.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:20.757370 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T04:30:20.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:20.757407 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T04:30:20.807 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:20.808050 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T04:30:21.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145624 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-1" 2022-03-08T04:30:21.147 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145684 I | op-osd: OSD PVC "smithi018-lv-1-data-09gnbf" already exists 2022-03-08T04:30:21.147 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145710 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-2" 2022-03-08T04:30:21.147 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145737 I | op-osd: OSD PVC "smithi018-lv-2-data-0rv5c5" already exists 2022-03-08T04:30:21.147 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145764 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-3" 2022-03-08T04:30:21.148 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145788 I | op-osd: OSD PVC "smithi018-lv-3-data-0fzdgm" already exists 2022-03-08T04:30:21.148 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145807 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-4" 2022-03-08T04:30:21.148 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145837 I | op-osd: OSD PVC "smithi018-lv-4-data-0srk9j" already exists 2022-03-08T04:30:21.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145855 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi084-lv-2" 2022-03-08T04:30:21.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145878 I | op-osd: OSD PVC "smithi084-lv-2-data-0t44bn" already exists 2022-03-08T04:30:21.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145898 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi084-lv-3" 2022-03-08T04:30:21.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.145952 I | op-osd: OSD PVC "smithi084-lv-3-data-0lnxp4" already exists 2022-03-08T04:30:21.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146027 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi084-lv-4" 2022-03-08T04:30:21.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146070 I | op-osd: OSD PVC "smithi084-lv-4-data-0pxmvd" already exists 2022-03-08T04:30:21.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146114 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi105-lv-1" 2022-03-08T04:30:21.152 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146189 I | op-osd: OSD PVC "smithi105-lv-1-data-0pnkl2" already exists 2022-03-08T04:30:21.152 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146208 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi105-lv-2" 2022-03-08T04:30:21.152 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146243 I | op-osd: OSD PVC "smithi105-lv-2-data-04z6t2" already exists 2022-03-08T04:30:21.152 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146267 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi105-lv-3" 2022-03-08T04:30:21.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146291 I | op-osd: OSD PVC "smithi105-lv-3-data-0gqb66" already exists 2022-03-08T04:30:21.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146307 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi105-lv-4" 2022-03-08T04:30:21.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146329 I | op-osd: OSD PVC "smithi105-lv-4-data-066ggl" already exists 2022-03-08T04:30:21.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.146348 I | op-osd: creating 1 new PVCs for device set "smithi084-lv-1" 2022-03-08T04:30:21.342 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.342511 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-08T04:30:21.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.342837 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-08T04:30:21.615 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.615971 I | op-osd: successfully provisioned PVC "smithi084-lv-1-data-0rhv8g" 2022-03-08T04:30:21.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680274 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-09gnbf" 2022-03-08T04:30:21.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680293 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-1-data-09gnbf" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680297 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:30:21.683 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680300 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-2-data-0rv5c5" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.683 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680303 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:30:21.683 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680306 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-3-data-0fzdgm" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.683 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680309 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0srk9j" 2022-03-08T04:30:21.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680311 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-4-data-0srk9j" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680314 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-2-data-0t44bn" 2022-03-08T04:30:21.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680317 I | op-osd: skipping OSD prepare job creation for PVC "smithi084-lv-2-data-0t44bn" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680320 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:30:21.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680324 I | op-osd: skipping OSD prepare job creation for PVC "smithi084-lv-3-data-0lnxp4" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680329 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:30:21.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680334 I | op-osd: skipping OSD prepare job creation for PVC "smithi084-lv-4-data-0pxmvd" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680338 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:30:21.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680341 I | op-osd: skipping OSD prepare job creation for PVC "smithi105-lv-1-data-0pnkl2" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680344 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-2-data-04z6t2" 2022-03-08T04:30:21.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680347 I | op-osd: skipping OSD prepare job creation for PVC "smithi105-lv-2-data-04z6t2" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680350 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-3-data-0gqb66" 2022-03-08T04:30:21.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680353 I | op-osd: skipping OSD prepare job creation for PVC "smithi105-lv-3-data-0gqb66" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680356 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-4-data-066ggl" 2022-03-08T04:30:21.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680359 I | op-osd: skipping OSD prepare job creation for PVC "smithi105-lv-4-data-066ggl" because OSD daemon using the PVC already exists 2022-03-08T04:30:21.688 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.680362 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-1-data-0rhv8g" 2022-03-08T04:30:21.750 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.750145 I | op-mgr: successful modules: prometheus 2022-03-08T04:30:21.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.752292 I | op-mgr: successful modules: dashboard 2022-03-08T04:30:21.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:21.752720 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T04:30:22.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.161726 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-08T04:30:22.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.173609 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-08T04:30:22.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.173626 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T04:30:22.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.364864 I | op-osd: started OSD provisioning job for PVC "smithi084-lv-1-data-0rhv8g" 2022-03-08T04:30:22.365 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.364895 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T04:30:22.365 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.364902 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T04:30:22.742 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.742727 I | op-osd: OSD orchestration status for PVC smithi084-lv-1-data-0rhv8g is "starting" 2022-03-08T04:30:22.844 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.844253 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:22.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.849989 I | op-osd: updating OSD 1 on PVC "smithi084-lv-2-data-0t44bn" 2022-03-08T04:30:22.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:22.850017 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-2-data-0t44bn" 2022-03-08T04:30:23.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.094888 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:23.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.100832 I | op-osd: updating OSD 10 on PVC "smithi105-lv-3-data-0gqb66" 2022-03-08T04:30:23.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.100873 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-3-data-0gqb66" 2022-03-08T04:30:23.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.155324 I | op-mon: mons running: [c a b] 2022-03-08T04:30:23.269 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.269427 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:23.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.276343 I | op-osd: updating OSD 11 on PVC "smithi105-lv-4-data-066ggl" 2022-03-08T04:30:23.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.276385 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-4-data-066ggl" 2022-03-08T04:30:23.352 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.352173 I | op-mgr: successful modules: orchestrator modules 2022-03-08T04:30:23.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.465635 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:23.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.471748 I | op-osd: updating OSD 2 on PVC "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:30:23.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.471779 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:30:23.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.564580 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:30:23.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:23.851893 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-08T04:30:24.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:24.284412 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-08T04:30:24.285 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:24.284450 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T04:30:24.342 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:24.342459 I | op-mon: mons running: [c a b] 2022-03-08T04:30:24.571 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:24.571185 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:24.650 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:24.650258 I | op-osd: updating OSD 3 on PVC "smithi018-lv-1-data-09gnbf" 2022-03-08T04:30:24.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:24.650292 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-09gnbf" 2022-03-08T04:30:24.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:24.876928 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:30:25.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:25.250709 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-08T04:30:25.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:25.561127 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:25.657 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:25.656854 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-08T04:30:25.657 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:25.656882 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T04:30:25.692 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:25.692341 I | op-mon: mons running: [c a b] 2022-03-08T04:30:25.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:25.888543 I | op-osd: updating OSD 4 on PVC "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:30:25.889 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:25.888568 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:30:26.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:26.094785 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:30:26.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:26.094814 I | op-mon: mons created: 3 2022-03-08T04:30:26.498 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:26.498411 I | op-mon: waiting for mon quorum with [c a b] 2022-03-08T04:30:26.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:26.565805 I | op-mon: mons running: [c a b] 2022-03-08T04:30:26.575 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:26.573497 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:26.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:26.651268 I | op-osd: updating OSD 5 on PVC "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:30:26.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:26.651309 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:30:26.987 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:26.987821 I | op-mon: Monitors in quorum: [a b c] 2022-03-08T04:30:26.988 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:26.987866 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-08T04:30:27.355 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:27.355377 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:27.450 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:27.450681 I | op-osd: updating OSD 6 on PVC "smithi018-lv-4-data-0srk9j" 2022-03-08T04:30:27.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:27.450719 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0srk9j" 2022-03-08T04:30:27.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:27.469961 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-08T04:30:27.988 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:27.988613 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-08T04:30:28.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:28.171742 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:28.255 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:28.255279 I | op-osd: updating OSD 7 on PVC "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:30:28.255 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:28.255314 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:30:28.410 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:28.410630 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-08T04:30:28.848 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:28.848213 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-08T04:30:28.848 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:28.848243 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-08T04:30:28.956 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:28.955602 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:29.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:29.050447 I | op-osd: updating OSD 8 on PVC "smithi105-lv-2-data-04z6t2" 2022-03-08T04:30:29.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:29.050480 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-2-data-04z6t2" 2022-03-08T04:30:29.297 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:29.297990 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-08T04:30:29.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:29.746045 I | cephclient: successfully enabled msgr2 protocol 2022-03-08T04:30:29.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:29.746085 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-08T04:30:29.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:29.746094 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-08T04:30:29.756 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:29.756500 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:29.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:29.851726 I | op-osd: updating OSD 9 on PVC "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:30:29.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:29.851768 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:30:30.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:30.188832 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-08T04:30:30.224 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:30.224947 I | op-mgr: start running mgr 2022-03-08T04:30:30.225 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:30.224987 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-08T04:30:30.726 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:30.726412 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-08T04:30:30.855 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:30.855032 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-08T04:30:31.089 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.089809 I | op-mgr: setting services to point to mgr "a" 2022-03-08T04:30:31.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.227682 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-08T04:30:31.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.227818 I | op-mgr: successful modules: balancer 2022-03-08T04:30:31.242 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.242806 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-08T04:30:31.242 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.242841 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-08T04:30:31.288 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.288402 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-08T04:30:31.300 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299096 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-1" 2022-03-08T04:30:31.301 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299129 I | op-osd: OSD PVC "smithi018-lv-1-data-09gnbf" already exists 2022-03-08T04:30:31.301 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299142 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-2" 2022-03-08T04:30:31.301 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299153 I | op-osd: OSD PVC "smithi018-lv-2-data-0rv5c5" already exists 2022-03-08T04:30:31.301 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299162 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-3" 2022-03-08T04:30:31.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299177 I | op-osd: OSD PVC "smithi018-lv-3-data-0fzdgm" already exists 2022-03-08T04:30:31.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299185 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-4" 2022-03-08T04:30:31.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299196 I | op-osd: OSD PVC "smithi018-lv-4-data-0srk9j" already exists 2022-03-08T04:30:31.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299202 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi084-lv-2" 2022-03-08T04:30:31.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299211 I | op-osd: OSD PVC "smithi084-lv-2-data-0t44bn" already exists 2022-03-08T04:30:31.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299219 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi084-lv-3" 2022-03-08T04:30:31.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299228 I | op-osd: OSD PVC "smithi084-lv-3-data-0lnxp4" already exists 2022-03-08T04:30:31.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299234 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi084-lv-4" 2022-03-08T04:30:31.304 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299246 I | op-osd: OSD PVC "smithi084-lv-4-data-0pxmvd" already exists 2022-03-08T04:30:31.304 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299254 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi105-lv-1" 2022-03-08T04:30:31.304 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299263 I | op-osd: OSD PVC "smithi105-lv-1-data-0pnkl2" already exists 2022-03-08T04:30:31.304 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299269 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi105-lv-2" 2022-03-08T04:30:31.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299278 I | op-osd: OSD PVC "smithi105-lv-2-data-04z6t2" already exists 2022-03-08T04:30:31.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299289 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi105-lv-3" 2022-03-08T04:30:31.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299300 I | op-osd: OSD PVC "smithi105-lv-3-data-0gqb66" already exists 2022-03-08T04:30:31.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299307 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi105-lv-4" 2022-03-08T04:30:31.306 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299316 I | op-osd: OSD PVC "smithi105-lv-4-data-066ggl" already exists 2022-03-08T04:30:31.306 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299324 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi084-lv-1" 2022-03-08T04:30:31.306 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.299336 I | op-osd: OSD PVC "smithi084-lv-1-data-0rhv8g" already exists 2022-03-08T04:30:31.476 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475155 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-09gnbf" 2022-03-08T04:30:31.476 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475179 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-1-data-09gnbf" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475187 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:30:31.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475192 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-2-data-0rv5c5" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475196 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:30:31.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475201 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-3-data-0fzdgm" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475205 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0srk9j" 2022-03-08T04:30:31.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475210 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-4-data-0srk9j" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475215 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-2-data-0t44bn" 2022-03-08T04:30:31.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475219 I | op-osd: skipping OSD prepare job creation for PVC "smithi084-lv-2-data-0t44bn" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475224 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:30:31.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475228 I | op-osd: skipping OSD prepare job creation for PVC "smithi084-lv-3-data-0lnxp4" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475235 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:30:31.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475239 I | op-osd: skipping OSD prepare job creation for PVC "smithi084-lv-4-data-0pxmvd" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475244 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:30:31.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475249 I | op-osd: skipping OSD prepare job creation for PVC "smithi105-lv-1-data-0pnkl2" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475253 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-2-data-04z6t2" 2022-03-08T04:30:31.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475257 I | op-osd: skipping OSD prepare job creation for PVC "smithi105-lv-2-data-04z6t2" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475262 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-3-data-0gqb66" 2022-03-08T04:30:31.482 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475268 I | op-osd: skipping OSD prepare job creation for PVC "smithi105-lv-3-data-0gqb66" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.482 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475274 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-4-data-066ggl" 2022-03-08T04:30:31.482 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475279 I | op-osd: skipping OSD prepare job creation for PVC "smithi105-lv-4-data-066ggl" because OSD daemon using the PVC already exists 2022-03-08T04:30:31.483 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.475284 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-1-data-0rhv8g" 2022-03-08T04:30:31.520 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.520111 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi084-lv-1-data-0rhv8g. Status={Conditions:[] StartTime:2022-03-08 04:30:22 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-08T04:30:31.520 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.520145 I | op-osd: started OSD provisioning job for PVC "smithi084-lv-1-data-0rhv8g" 2022-03-08T04:30:31.521 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.520160 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-08T04:30:31.521 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.520171 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-08T04:30:31.662 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.660265 I | op-osd: OSD orchestration status for PVC smithi084-lv-1-data-0rhv8g is "starting" 2022-03-08T04:30:31.762 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.761726 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:31.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.803840 I | op-osd: updating OSD 1 on PVC "smithi084-lv-2-data-0t44bn" 2022-03-08T04:30:31.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:31.803860 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-2-data-0t44bn" 2022-03-08T04:30:32.355 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:32.354991 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:32.449 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:32.449479 I | op-osd: updating OSD 10 on PVC "smithi105-lv-3-data-0gqb66" 2022-03-08T04:30:32.449 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:32.449510 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-3-data-0gqb66" 2022-03-08T04:30:32.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:32.760574 I | op-mgr: successful modules: dashboard 2022-03-08T04:30:32.764 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:32.765096 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-08T04:30:32.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:32.765225 I | op-mgr: successful modules: prometheus 2022-03-08T04:30:33.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:33.157593 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:33.251 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:33.251664 I | op-osd: updating OSD 11 on PVC "smithi105-lv-4-data-066ggl" 2022-03-08T04:30:33.251 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:33.251688 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-4-data-066ggl" 2022-03-08T04:30:33.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:33.967509 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:34.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:34.050832 I | op-osd: updating OSD 2 on PVC "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:30:34.051 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:34.050868 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-4-data-0pxmvd" 2022-03-08T04:30:34.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:34.776588 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:34.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:34.849438 I | op-osd: updating OSD 3 on PVC "smithi018-lv-1-data-09gnbf" 2022-03-08T04:30:34.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:34.849459 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-09gnbf" 2022-03-08T04:30:34.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:34.893843 I | op-mgr: successful modules: orchestrator modules 2022-03-08T04:30:35.558 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:35.558660 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:35.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:35.651119 I | op-osd: updating OSD 4 on PVC "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:30:35.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:35.651146 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0rv5c5" 2022-03-08T04:30:36.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:36.360074 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:36.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:36.478654 I | op-osd: updating OSD 5 on PVC "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:30:36.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:36.478686 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-3-data-0lnxp4" 2022-03-08T04:30:37.158 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:37.158595 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:37.278 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:37.278087 I | op-osd: updating OSD 6 on PVC "smithi018-lv-4-data-0srk9j" 2022-03-08T04:30:37.278 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:37.278117 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0srk9j" 2022-03-08T04:30:37.958 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:37.958492 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:38.051 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:38.051823 I | op-osd: updating OSD 7 on PVC "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:30:38.052 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:38.051856 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0fzdgm" 2022-03-08T04:30:38.766 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:38.766365 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:38.850 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:38.850165 I | op-osd: updating OSD 8 on PVC "smithi105-lv-2-data-04z6t2" 2022-03-08T04:30:38.850 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:38.850196 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-2-data-04z6t2" 2022-03-08T04:30:38.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:38.946911 I | op-osd: OSD orchestration status for node smithi084-lv-1-data-0rhv8g is "orchestrating" 2022-03-08T04:30:39.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:39.348405 I | op-osd: OSD orchestration status for PVC smithi084-lv-1-data-0rhv8g is "orchestrating" 2022-03-08T04:30:39.457 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:39.457355 I | op-osd: OSD orchestration status for node smithi084-lv-1-data-0rhv8g is "orchestrating" 2022-03-08T04:30:39.460 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:39.457680 I | op-osd: OSD orchestration status for PVC smithi084-lv-1-data-0rhv8g is "orchestrating" 2022-03-08T04:30:39.558 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:39.558109 I | op-osd: skipping osd checks for ok-to-stop 2022-03-08T04:30:39.652 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:39.652620 I | op-osd: updating OSD 9 on PVC "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:30:39.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:39.652654 I | op-osd: OSD will have its main bluestore block on "smithi105-lv-1-data-0pnkl2" 2022-03-08T04:30:44.768 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T04:30:44.769 INFO:teuthology.orchestra.run.smithi018.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T04:30:44.770 INFO:teuthology.orchestra.run.smithi018.stdout:waiting for osd to come back up 2022-03-08T04:30:44.771 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-08T04:30:44.771 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T04:30:44.771 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-08T04:30:44.843 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T04:30:45.303 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=11 2022-03-08T04:30:45.304 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 30s 2022-03-08T04:30:45.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:45.479826 I | op-osd: OSD orchestration status for PVC smithi084-lv-1-data-0rhv8g is "completed" 2022-03-08T04:30:45.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:45.479988 I | op-osd: creating OSD 0 on PVC "smithi084-lv-1-data-0rhv8g" 2022-03-08T04:30:45.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:45.480010 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-1-data-0rhv8g" 2022-03-08T04:30:45.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:45.480054 I | op-osd: OSD orchestration status for PVC smithi084-lv-1-data-0rhv8g is "completed" 2022-03-08T04:30:45.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:45.480103 I | op-osd: creating OSD 0 on PVC "smithi084-lv-1-data-0rhv8g" 2022-03-08T04:30:45.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:45.480127 I | op-osd: OSD will have its main bluestore block on "smithi084-lv-1-data-0rhv8g" 2022-03-08T04:30:45.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:45.627202 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 0 on PVC "smithi084-lv-1-data-0rhv8g" LastHeartbeatTime:2022-03-08 04:30:45.582860151 +0000 UTC m=+542.726927223 LastTransitionTime:2022-03-08 04:30:45.582859998 +0000 UTC m=+542.726927122}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:30:45.717 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:45.716731 E | op-osd: failed to create OSD 0 on PVC "smithi084-lv-1-data-0rhv8g": failed to create deployment for OSD 0 on PVC "smithi084-lv-1-data-0rhv8g": deployments.apps "rook-ceph-osd-0" already exists 2022-03-08T04:30:45.924 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:45.922207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-08T04:30:45.924 INFO:tasks.rook.operator.smithi018.stdout:failed to create OSD 0 on PVC "smithi084-lv-1-data-0rhv8g": failed to create deployment for OSD 0 on PVC "smithi084-lv-1-data-0rhv8g": deployments.apps "rook-ceph-osd-0" already exists 2022-03-08T04:30:46.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:46.215238 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-08T04:30:46.216 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:30:46.215261 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-08T04:31:15.305 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 11 '!=' 12 ']' 2022-03-08T04:31:15.305 INFO:teuthology.orchestra.run.smithi018.stderr:+ echo 'waiting for osd to come back up' 2022-03-08T04:31:15.306 INFO:teuthology.orchestra.run.smithi018.stdout:waiting for osd to come back up 2022-03-08T04:31:15.306 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-08T04:31:15.307 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-08T04:31:15.307 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-08T04:31:15.380 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-08T04:31:15.895 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=12 2022-03-08T04:31:15.896 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 30s 2022-03-08T04:31:45.897 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 12 '!=' 12 ']' 2022-03-08T04:31:45.899 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-08T04:31:45.918 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch status 2022-03-08T04:31:46.971 INFO:teuthology.orchestra.run.smithi018.stdout:Backend: rook 2022-03-08T04:31:46.971 INFO:teuthology.orchestra.run.smithi018.stdout:Available: Yes 2022-03-08T04:31:46.996 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ps 2022-03-08T04:31:49.120 INFO:teuthology.orchestra.run.smithi018.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-08T04:31:49.121 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi018 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.121 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi084 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-08T04:31:49.121 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi105 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T04:31:49.122 INFO:teuthology.orchestra.run.smithi018.stdout:mgr.a smithi018 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-08T04:31:49.122 INFO:teuthology.orchestra.run.smithi018.stdout:mon.a smithi105 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T04:31:49.122 INFO:teuthology.orchestra.run.smithi018.stdout:mon.b smithi084 running (6m) 0s ago 7m - - 4b871cde5e1a 2022-03-08T04:31:49.122 INFO:teuthology.orchestra.run.smithi018.stdout:mon.c smithi018 running (6m) 0s ago 6m - - 4b871cde5e1a 2022-03-08T04:31:49.123 INFO:teuthology.orchestra.run.smithi018.stdout:osd.0 smithi084 running (64s) 0s ago 64s - - 4b871cde5e1a 2022-03-08T04:31:49.123 INFO:teuthology.orchestra.run.smithi018.stdout:osd.1 smithi084 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.123 INFO:teuthology.orchestra.run.smithi018.stdout:osd.10 smithi105 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.123 INFO:teuthology.orchestra.run.smithi018.stdout:osd.11 smithi105 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.124 INFO:teuthology.orchestra.run.smithi018.stdout:osd.2 smithi084 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.124 INFO:teuthology.orchestra.run.smithi018.stdout:osd.3 smithi018 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.124 INFO:teuthology.orchestra.run.smithi018.stdout:osd.4 smithi018 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.124 INFO:teuthology.orchestra.run.smithi018.stdout:osd.5 smithi084 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.125 INFO:teuthology.orchestra.run.smithi018.stdout:osd.6 smithi018 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.125 INFO:teuthology.orchestra.run.smithi018.stdout:osd.7 smithi018 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.125 INFO:teuthology.orchestra.run.smithi018.stdout:osd.8 smithi105 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.125 INFO:teuthology.orchestra.run.smithi018.stdout:osd.9 smithi105 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-08T04:31:49.140 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls 2022-03-08T04:31:50.655 INFO:teuthology.orchestra.run.smithi018.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-08T04:31:50.655 INFO:teuthology.orchestra.run.smithi018.stdout:crash 3/3 0s ago 5m * 2022-03-08T04:31:50.655 INFO:teuthology.orchestra.run.smithi018.stdout:mgr 1/1 0s ago 5m count:1 2022-03-08T04:31:50.656 INFO:teuthology.orchestra.run.smithi018.stdout:mon 3/3 0s ago 7m count:3 2022-03-08T04:31:50.656 INFO:teuthology.orchestra.run.smithi018.stdout:osd 24 0s ago 3m 2022-03-08T04:31:50.656 INFO:teuthology.orchestra.run.smithi018.stdout:osd.all-available-devices 0 0s ago - * 2022-03-08T04:31:50.701 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch host ls 2022-03-08T04:31:51.220 INFO:teuthology.orchestra.run.smithi018.stdout:HOST ADDR LABELS STATUS 2022-03-08T04:31:51.221 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 172.21.15.18/smithi018 2022-03-08T04:31:51.221 INFO:teuthology.orchestra.run.smithi018.stdout:smithi084 172.21.15.84/smithi084 2022-03-08T04:31:51.221 INFO:teuthology.orchestra.run.smithi018.stdout:smithi105 172.21.15.105/smithi105 2022-03-08T04:31:51.221 INFO:teuthology.orchestra.run.smithi018.stdout:3 hosts in cluster 2022-03-08T04:31:51.245 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch device ls 2022-03-08T04:31:51.796 INFO:teuthology.orchestra.run.smithi018.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-08T04:31:51.797 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T04:31:51.797 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T04:31:51.797 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T04:31:51.797 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T04:31:51.798 INFO:teuthology.orchestra.run.smithi018.stdout:smithi084 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T04:31:51.798 INFO:teuthology.orchestra.run.smithi018.stdout:smithi084 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T04:31:51.798 INFO:teuthology.orchestra.run.smithi018.stdout:smithi084 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T04:31:51.799 INFO:teuthology.orchestra.run.smithi018.stdout:smithi084 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T04:31:51.799 INFO:teuthology.orchestra.run.smithi018.stdout:smithi105 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-08T04:31:51.799 INFO:teuthology.orchestra.run.smithi018.stdout:smithi105 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-08T04:31:51.800 INFO:teuthology.orchestra.run.smithi018.stdout:smithi105 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-08T04:31:51.800 INFO:teuthology.orchestra.run.smithi018.stdout:smithi105 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-08T04:31:51.820 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch apply rgw foo 2022-03-08T04:31:52.357 INFO:teuthology.orchestra.run.smithi018.stdout:Created 2022-03-08T04:31:52.359 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:31:52.359991 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-08T04:31:52.377 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch apply mds foo 2022-03-08T04:31:52.527 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:31:52.527964 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:31:52.528 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:31:52.528044 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T04:31:52.914 INFO:teuthology.orchestra.run.smithi018.stdout:Created 2022-03-08T04:31:52.914 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:31:52.914995 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T04:31:52.936 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch apply rbd-mirror 2022-03-08T04:31:52.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:31:52.941975 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:31:52.945 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:31:52.946082 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:31:52.945 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:31:52.946138 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T04:31:53.713 INFO:teuthology.orchestra.run.smithi018.stdout:Success 2022-03-08T04:31:53.718 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:31:53.719424 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:31:53.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:31:53.719489 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-08T04:31:53.739 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch apply nfs foo --port 12777 2022-03-08T04:31:57.287 INFO:teuthology.orchestra.run.smithi018.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-08T04:31:57.288 INFO:teuthology.orchestra.run.smithi018.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-08T04:31:57.312 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-08T04:31:57.324 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:31:57.973 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:31:57.973 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:37.000000Z", "last_refresh": "2022-03-08T04:31:57.791519Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:31:57.791519Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:31:57.791519Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:31:57.791519Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:31:57.791519Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:31:57.791519Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-08T04:31:57.791519Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:31:57.791519Z", "running": 0, "size": 1}}] 2022-03-08T04:31:57.998 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch rm mds.foo 2022-03-08T04:31:58.798 INFO:teuthology.orchestra.run.smithi018.stdout:Removed cephfilesystems/foo 2022-03-08T04:31:58.819 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch rm rbd-mirror 2022-03-08T04:31:59.416 INFO:teuthology.orchestra.run.smithi018.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-08T04:31:59.440 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch rm rgw.foo 2022-03-08T04:32:00.043 INFO:teuthology.orchestra.run.smithi018.stdout:Removed cephobjectstores/foo 2022-03-08T04:32:00.088 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:32:00.944 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:00.945400 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T04:32:01.014 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:32:01.015 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:37.000000Z", "last_refresh": "2022-03-08T04:32:00.836322Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:00.836322Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:32:00.836322Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:32:00.836322Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:32:00.836322Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:32:00.836322Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:00.836322Z", "running": 0, "size": 1}}] 2022-03-08T04:32:01.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:01.348413 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-08T04:32:01.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:01.348445 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-08T04:32:01.349 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:01.348473 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-08T04:32:02.267 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:02.267710 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-08T04:32:02.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:02.587696 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-08T04:32:02.717 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:02.718254 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi084": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:32:02.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:02.834007 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi084": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:32:02.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:02.881867 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T04:32:02.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:02.892465 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-08T04:32:04.632 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:04.632095 I | ceph-object-controller: reconciling object store deployments 2022-03-08T04:32:04.703 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:04.703417 I | ceph-object-controller: ceph object store gateway service running at 10.106.56.54 2022-03-08T04:32:04.703 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:04.703451 I | ceph-object-controller: reconciling object store pools 2022-03-08T04:32:04.825 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:04.814961 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-08T04:32:05.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:05.119096 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-08T04:32:05.475 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:05.475563 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-08T04:32:06.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:06.716409 I | op-mds: setting mds config flags 2022-03-08T04:32:06.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:06.716453 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T04:32:07.606 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:07.607144 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T04:32:07.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:07.670085 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-08T04:32:08.459 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:08.459788 I | op-mds: setting mds config flags 2022-03-08T04:32:08.459 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:08.459815 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T04:32:11.045 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:32:11.816 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:32:11.816 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:37.000000Z", "last_refresh": "2022-03-08T04:32:11.598226Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:11.598226Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:32:11.598226Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:32:11.598226Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:32:11.598226Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:32:11.598226Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:11.598226Z", "running": 0, "size": 1}}] 2022-03-08T04:32:11.842 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:11.843241 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-08T04:32:13.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:13.715440 I | ceph-file-controller: creating filesystem "foo" 2022-03-08T04:32:15.035 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:15.035923 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-08T04:32:15.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:15.036230 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-08T04:32:16.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:16.155573 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-08T04:32:16.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:16.155717 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-08T04:32:16.168 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:16.160749 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-08T04:32:18.117 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:18.117301 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-08T04:32:18.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:18.156005 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:20.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:20.292193 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-08T04:32:21.074 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:21.074017 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:21.066+0000 7f9a57071380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:21.074 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:21.066+0000 7f9a57071380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:21.074 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:21.066+0000 7f9a57071380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:21.075 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:21.143 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:21.144463 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:21.699 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:21.699386 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:21.690+0000 7fdc241ac380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:21.699 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:21.690+0000 7fdc241ac380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:21.700 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:21.690+0000 7fdc241ac380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:21.700 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:21.738 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:21.738351 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:21.848 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:32:22.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:22.360564 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:22.350+0000 7ff72f5b1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:22.361 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:22.350+0000 7ff72f5b1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:22.361 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:22.350+0000 7ff72f5b1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:22.361 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:22.416 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:22.416555 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:22.579 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:32:22.579 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:37.000000Z", "last_refresh": "2022-03-08T04:32:22.402387Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:22.402387Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:32:22.402387Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:32:22.402387Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:32:22.402387Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:32:22.402387Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:22.402387Z", "running": 0, "size": 1}}] 2022-03-08T04:32:23.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:23.002535 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:22.994+0000 7fac5e818380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:23.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:22.994+0000 7fac5e818380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:23.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:22.994+0000 7fac5e818380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:23.003 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:23.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:23.018555 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:23.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:23.589475 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:23.578+0000 7ff0ad1ac380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:23.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:23.578+0000 7ff0ad1ac380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:23.590 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:23.578+0000 7ff0ad1ac380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:23.590 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:23.781 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:23.781661 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:24.445 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:24.445961 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-08T04:32:24.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:24.497032 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:24.486+0000 7fa434ab0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:24.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:24.486+0000 7fa434ab0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:24.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:24.486+0000 7fa434ab0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:24.498 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:24.526 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:24.527293 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:25.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:25.107944 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:25.098+0000 7fe7e652f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:25.108 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:25.098+0000 7fe7e652f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:25.108 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:25.098+0000 7fe7e652f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:25.109 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:25.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:25.133003 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-08T04:32:25.141 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:25.142534 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:26.197 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:26.196989 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:26.186+0000 7f6a6a121380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:26.197 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:26.186+0000 7f6a6a121380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:26.197 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:26.186+0000 7f6a6a121380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:26.198 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:26.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:26.247441 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:26.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:26.808850 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:26.798+0000 7f0e27c76380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:26.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:26.798+0000 7f0e27c76380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:26.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:26.798+0000 7f0e27c76380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:26.810 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:27.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:27.485720 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:27.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:27.800003 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-08T04:32:27.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:27.800023 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-08T04:32:28.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:28.092180 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:28.082+0000 7fd14744e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:28.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:28.082+0000 7fd14744e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:28.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:28.082+0000 7fd14744e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:28.093 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:28.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:28.874925 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:32:28.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:28.879989 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:28.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:28.882143 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-08T04:32:29.673 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:29.673390 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T04:32:30.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:30.174097 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T04:32:30.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:30.174117 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-08T04:32:30.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:30.174121 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-08T04:32:30.935 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:30.934968 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-08T04:32:31.283 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:31.283948 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-08T04:32:31.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:31.757137 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-08T04:32:31.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:31.757175 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-08T04:32:31.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:31.757184 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-08T04:32:32.322 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:32.322337 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-08T04:32:32.322 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:32.322371 I | ceph-file-controller: downing filesystem "foo" 2022-03-08T04:32:32.602 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:32:32.970 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:32.970146 I | ceph-file-controller: downed filesystem "foo" 2022-03-08T04:32:33.224 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:33.225501 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:33.388 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:32:33.388 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:32:33.216601Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:33.216601Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:32:33.216601Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:32:33.216601Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:32:33.216601Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:32:33.216601Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:33.216601Z", "running": 0, "size": 1}}] 2022-03-08T04:32:33.767 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:33.767565 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:33.758+0000 7f08485ad380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:33.768 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:33.758+0000 7f08485ad380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:33.768 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:33.758+0000 7f08485ad380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:33.768 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:36.904 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:36.904495 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-08T04:32:36.905 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:36.904511 I | cephclient: purging pool "foo-metadata" (id=8) 2022-03-08T04:32:39.959 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:39.959430 I | cephclient: purge completed for pool "foo-metadata" 2022-03-08T04:32:40.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:40.414994 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-08T04:32:40.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:40.415021 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-08T04:32:42.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:42.390764 I | cephclient: purge completed for pool "foo-data0" 2022-03-08T04:32:42.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:42.390855 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-08T04:32:42.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:42.451574 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-08T04:32:42.458 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:42.458903 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-08T04:32:42.462 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:42.459697 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-08T04:32:42.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:42.468227 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-08T04:32:43.412 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:32:44.044 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:44.044110 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:32:44.282 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:32:44.283 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:32:44.036048Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:32:44.036048Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:32:44.036048Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:32:44.036048Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:32:44.036048Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:44.036048Z", "running": 0, "size": 1}}] 2022-03-08T04:32:44.617 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:44.617888 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:32:44.610+0000 7f8196020380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:32:44.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:44.610+0000 7f8196020380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:32:44.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:32:44.610+0000 7f8196020380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:32:44.618 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:32:46.065 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:32:46.065835 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi105": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:32:54.306 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:32:55.014 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:32:55.014 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:32:54.840341Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:32:54.840341Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:32:54.840341Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:32:54.840341Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:32:54.840341Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:32:54.840341Z", "running": 0, "size": 1}}] 2022-03-08T04:33:05.041 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:33:05.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:33:05.105872 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:33:05.724 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:33:05.724565 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:33:05.714+0000 7fc822c5f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:33:05.724 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:33:05.714+0000 7fc822c5f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:33:05.725 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:33:05.714+0000 7fc822c5f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:33:05.725 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:33:05.729 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:33:05.730 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:33:05.556438Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:33:05.556438Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:33:05.556438Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:33:05.556438Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:33:05.556438Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:33:05.556438Z", "running": 0, "size": 1}}] 2022-03-08T04:33:15.757 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:33:16.457 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:33:16.457 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:33:16.271939Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:33:16.271939Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:33:16.271939Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:33:16.271939Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:33:16.271939Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:33:16.271939Z", "running": 0, "size": 1}}] 2022-03-08T04:33:26.481 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:33:27.251 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:33:27.251 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:33:27.040974Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:33:27.040974Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:33:27.040974Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:33:27.040974Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:33:27.040974Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:33:27.040974Z", "running": 0, "size": 1}}] 2022-03-08T04:33:37.280 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:33:37.989 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:33:37.989 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:33:37.804666Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:33:37.804666Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:33:37.804666Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:33:37.804666Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:33:37.804666Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:33:37.804666Z", "running": 0, "size": 1}}] 2022-03-08T04:33:46.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:33:46.698351 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:33:47.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:33:47.407066 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:33:47.397+0000 7fd7843e0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:33:47.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:33:47.397+0000 7fd7843e0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:33:47.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:33:47.397+0000 7fd7843e0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:33:47.407 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:33:48.017 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:33:48.748 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:33:48.749 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:33:48.574433Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:33:48.574433Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:33:48.574433Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:33:48.574433Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:33:48.574433Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:33:48.574433Z", "running": 0, "size": 1}}] 2022-03-08T04:33:58.773 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:33:59.821 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:33:59.821 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:33:59.329717Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:33:59.329717Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:33:59.329717Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:33:59.329717Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:33:59.329717Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:33:59.329717Z", "running": 0, "size": 1}}] 2022-03-08T04:34:07.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:34:07.451688 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi084": the object has been modified; please apply your changes to the latest version and try again 2022-03-08T04:34:09.848 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:34:10.605 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:34:10.606 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:34:10.424085Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:34:10.424085Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:34:10.424085Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:34:10.424085Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:34:10.424085Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:34:10.424085Z", "running": 0, "size": 1}}] 2022-03-08T04:34:20.634 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:34:21.330 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:34:21.330 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:34:21.165190Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:34:21.165190Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:34:21.165190Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:34:21.165190Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:34:21.165190Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:34:21.165190Z", "running": 0, "size": 1}}] 2022-03-08T04:34:31.356 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:34:32.060 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:34:32.060 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:34:31.900129Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:34:31.900129Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:34:31.900129Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:34:31.900129Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:34:31.900129Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:34:31.900129Z", "running": 0, "size": 1}}] 2022-03-08T04:34:42.087 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:34:42.813 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:34:42.814 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:34:42.668598Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:34:42.668598Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:34:42.668598Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:34:42.668598Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:34:42.668598Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:34:42.668598Z", "running": 0, "size": 1}}] 2022-03-08T04:34:52.838 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:34:53.809 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:34:53.809 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:34:53.380902Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:34:53.380902Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:34:53.380902Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:34:53.380902Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:34:53.380902Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:34:53.380902Z", "running": 0, "size": 1}}] 2022-03-08T04:35:03.834 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:35:04.555 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:35:04.555 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:35:04.383217Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:35:04.383217Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:35:04.383217Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:35:04.383217Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:35:04.383217Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:35:04.383217Z", "running": 0, "size": 1}}] 2022-03-08T04:35:09.356 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:35:09.357345 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:35:10.006 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:35:10.006399 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:35:09.995+0000 7f53b040c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:35:10.006 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:35:09.995+0000 7f53b040c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:35:10.006 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:35:09.995+0000 7f53b040c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:35:10.007 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:35:14.581 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:35:15.299 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:35:15.299 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:35:15.110942Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:35:15.110942Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:35:15.110942Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:35:15.110942Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:35:15.110942Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:35:15.110942Z", "running": 0, "size": 1}}] 2022-03-08T04:35:25.325 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:35:26.110 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:35:26.111 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:35:25.892419Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:35:25.892419Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:35:25.892419Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:35:25.892419Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:35:25.892419Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:35:25.892419Z", "running": 0, "size": 1}}] 2022-03-08T04:35:36.137 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:35:36.877 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:35:36.878 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:35:36.705852Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:35:36.705852Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:35:36.705852Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:35:36.705852Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:35:36.705852Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:35:36.705852Z", "running": 0, "size": 1}}] 2022-03-08T04:35:46.901 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:35:47.744 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:35:47.744 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:35:47.522370Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:35:47.522370Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:35:47.522370Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:35:47.522370Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:35:47.522370Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:35:47.522370Z", "running": 0, "size": 1}}] 2022-03-08T04:35:57.771 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:35:58.481 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:35:58.481 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:35:58.305887Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:35:58.305887Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:35:58.305887Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:35:58.305887Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:35:58.305887Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:35:58.305887Z", "running": 0, "size": 1}}] 2022-03-08T04:36:08.509 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:36:09.219 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:36:09.219 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:36:09.044169Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:36:09.044169Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:36:09.044169Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:36:09.044169Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:36:09.044169Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:36:09.044169Z", "running": 0, "size": 1}}] 2022-03-08T04:36:19.245 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:36:20.036 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:36:20.036 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:36:19.818967Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:36:19.818967Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:36:19.818967Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:36:19.818967Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:36:19.818967Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:36:19.818967Z", "running": 0, "size": 1}}] 2022-03-08T04:36:30.097 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:36:30.924 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:36:30.925 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:36:30.767244Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:36:30.767244Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:36:30.767244Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:36:30.767244Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:36:30.767244Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:36:30.767244Z", "running": 0, "size": 1}}] 2022-03-08T04:36:40.950 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:36:41.698 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:36:41.699 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:36:41.479982Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:36:41.479982Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:36:41.479982Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:36:41.479982Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:36:41.479982Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:36:41.479982Z", "running": 0, "size": 1}}] 2022-03-08T04:36:51.723 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:36:52.435 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:36:52.435 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:36:52.272265Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:36:52.272265Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:36:52.272265Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:36:52.272265Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:36:52.272265Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:36:52.272265Z", "running": 0, "size": 1}}] 2022-03-08T04:37:02.467 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:37:03.183 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:37:03.184 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:37:03.004277Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:37:03.004277Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:37:03.004277Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:37:03.004277Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:37:03.004277Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:37:03.004277Z", "running": 0, "size": 1}}] 2022-03-08T04:37:13.211 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:37:13.882 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:37:13.883 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:37:13.725163Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:37:13.725163Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:37:13.725163Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:37:13.725163Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:37:13.725163Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:37:13.725163Z", "running": 0, "size": 1}}] 2022-03-08T04:37:23.914 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:37:24.795 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:37:24.795 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:37:24.444629Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:37:24.444629Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:37:24.444629Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:37:24.444629Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:37:24.444629Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:37:24.444629Z", "running": 0, "size": 1}}] 2022-03-08T04:37:34.821 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:37:35.499 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:37:35.499 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:37:35.336933Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:37:35.336933Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:37:35.336933Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:37:35.336933Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:37:35.336933Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:37:35.336933Z", "running": 0, "size": 1}}] 2022-03-08T04:37:45.529 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:37:46.253 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:37:46.253 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:37:46.091890Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:37:46.091890Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:37:46.091890Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:37:46.091890Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:37:46.091890Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:37:46.091890Z", "running": 0, "size": 1}}] 2022-03-08T04:37:53.853 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:37:53.854423 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:37:54.598 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:37:54.598770 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:37:54.584+0000 7f61c0d04380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:37:54.598 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:37:54.584+0000 7f61c0d04380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:37:54.599 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:37:54.584+0000 7f61c0d04380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:37:54.599 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:37:56.279 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:37:57.018 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:37:57.018 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:37:56.847157Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:37:56.847157Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:37:56.847157Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:37:56.847157Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:37:56.847157Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:37:56.847157Z", "running": 0, "size": 1}}] 2022-03-08T04:38:07.047 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:38:07.823 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:38:07.824 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:38:07.611056Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:38:07.611056Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:38:07.611056Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:38:07.611056Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:38:07.611056Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:38:07.611056Z", "running": 0, "size": 1}}] 2022-03-08T04:38:17.850 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:38:18.543 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:38:18.544 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:38:18.379426Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:38:18.379426Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:38:18.379426Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:38:18.379426Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:38:18.379426Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:38:18.379426Z", "running": 0, "size": 1}}] 2022-03-08T04:38:28.569 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:38:29.310 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:38:29.311 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:38:29.128590Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:38:29.128590Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:38:29.128590Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:38:29.128590Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:38:29.128590Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:38:29.128590Z", "running": 0, "size": 1}}] 2022-03-08T04:38:39.337 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:38:40.075 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:38:40.075 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:38:39.906052Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:38:39.906052Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:38:39.906052Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:38:39.906052Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:38:39.906052Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:38:39.906052Z", "running": 0, "size": 1}}] 2022-03-08T04:38:50.114 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:38:50.843 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:38:50.843 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:38:50.650750Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:38:50.650750Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:38:50.650750Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:38:50.650750Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:38:50.650750Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:38:50.650750Z", "running": 0, "size": 1}}] 2022-03-08T04:39:00.868 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:39:01.610 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:39:01.611 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:39:01.407511Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:39:01.407511Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:39:01.407511Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:39:01.407511Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:39:01.407511Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:39:01.407511Z", "running": 0, "size": 1}}] 2022-03-08T04:39:11.636 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:39:12.518 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:39:12.519 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:39:12.191345Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:39:12.191345Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:39:12.191345Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:39:12.191345Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:39:12.191345Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:39:12.191345Z", "running": 0, "size": 1}}] 2022-03-08T04:39:22.543 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:39:23.266 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:39:23.266 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:39:23.067562Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:39:23.067562Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:39:23.067562Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:39:23.067562Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:39:23.067562Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:39:23.067562Z", "running": 0, "size": 1}}] 2022-03-08T04:39:33.289 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:39:34.063 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:39:34.064 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:39:33.860066Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:39:33.860066Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:39:33.860066Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:39:33.860066Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:39:33.860066Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:39:33.860066Z", "running": 0, "size": 1}}] 2022-03-08T04:39:44.095 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:39:45.117 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:39:45.118 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:39:44.664579Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:39:44.664579Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:39:44.664579Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:39:44.664579Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:39:44.664579Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:39:44.664579Z", "running": 0, "size": 1}}] 2022-03-08T04:39:55.144 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:39:55.923 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:39:55.924 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:39:55.678456Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:39:55.678456Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:39:55.678456Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:39:55.678456Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:39:55.678456Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:39:55.678456Z", "running": 0, "size": 1}}] 2022-03-08T04:40:05.946 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:40:06.687 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:40:06.687 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:40:06.506239Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:40:06.506239Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:40:06.506239Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:40:06.506239Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:40:06.506239Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:40:06.506239Z", "running": 0, "size": 1}}] 2022-03-08T04:40:16.710 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:40:17.403 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:40:17.404 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:40:17.229415Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:40:17.229415Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:40:17.229415Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:40:17.229415Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:40:17.229415Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:40:17.229415Z", "running": 0, "size": 1}}] 2022-03-08T04:40:27.428 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:40:28.235 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:40:28.236 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:40:27.975408Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:40:27.975408Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:40:27.975408Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:40:27.975408Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:40:27.975408Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:40:27.975408Z", "running": 0, "size": 1}}] 2022-03-08T04:40:38.262 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:40:38.996 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:40:38.997 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:40:38.796297Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:40:38.796297Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:40:38.796297Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:40:38.796297Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:40:38.796297Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:40:38.796297Z", "running": 0, "size": 1}}] 2022-03-08T04:40:49.020 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:40:49.706 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:40:49.707 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:40:49.540681Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:40:49.540681Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:40:49.540681Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:40:49.540681Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:40:49.540681Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:40:49.540681Z", "running": 0, "size": 1}}] 2022-03-08T04:40:59.731 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:41:00.475 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:41:00.475 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:41:00.278168Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:41:00.278168Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:41:00.278168Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:41:00.278168Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:41:00.278168Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:41:00.278168Z", "running": 0, "size": 1}}] 2022-03-08T04:41:10.508 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:41:11.206 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:41:11.207 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:41:11.024760Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:41:11.024760Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:41:11.024760Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:41:11.024760Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:41:11.024760Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:41:11.024760Z", "running": 0, "size": 1}}] 2022-03-08T04:41:21.235 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:41:21.959 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:41:21.960 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:41:21.783769Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:41:21.783769Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:41:21.783769Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:41:21.783769Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:41:21.783769Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:41:21.783769Z", "running": 0, "size": 1}}] 2022-03-08T04:41:31.985 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:41:32.704 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:41:32.704 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:41:32.527067Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:41:32.527067Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:41:32.527067Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:41:32.527067Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:41:32.527067Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:41:32.527067Z", "running": 0, "size": 1}}] 2022-03-08T04:41:42.729 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:41:43.456 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:41:43.456 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:41:43.279346Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:41:43.279346Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:41:43.279346Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:41:43.279346Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:41:43.279346Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:41:43.279346Z", "running": 0, "size": 1}}] 2022-03-08T04:41:53.482 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:41:57.047 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:41:57.048 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:41:56.879930Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:41:56.879930Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:41:56.879930Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:41:56.879930Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:41:56.879930Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:41:56.879930Z", "running": 0, "size": 1}}] 2022-03-08T04:42:07.075 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:42:07.756 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:42:07.757 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:42:07.586829Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:42:07.586829Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:42:07.586829Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:42:07.586829Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:42:07.586829Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:42:07.586829Z", "running": 0, "size": 1}}] 2022-03-08T04:42:17.784 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:42:18.501 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:42:18.502 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:42:18.333612Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:42:18.333612Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:42:18.333612Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:42:18.333612Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:42:18.333612Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:42:18.333612Z", "running": 0, "size": 1}}] 2022-03-08T04:42:28.529 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:42:29.226 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:42:29.226 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:42:29.061739Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:42:29.061739Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:42:29.061739Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:42:29.061739Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:42:29.061739Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:42:29.061739Z", "running": 0, "size": 1}}] 2022-03-08T04:42:39.252 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:42:40.032 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:42:40.032 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:42:39.868695Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:42:39.868695Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:42:39.868695Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:42:39.868695Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:42:39.868695Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:42:39.868695Z", "running": 0, "size": 1}}] 2022-03-08T04:42:50.063 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:42:50.791 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:42:50.792 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:42:50.617997Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:42:50.617997Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:42:50.617997Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:42:50.617997Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:42:50.617997Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:42:50.617997Z", "running": 0, "size": 1}}] 2022-03-08T04:43:00.819 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:43:01.569 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:43:01.570 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:43:01.360692Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:43:01.360692Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:43:01.360692Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:43:01.360692Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:43:01.360692Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:43:01.360692Z", "running": 0, "size": 1}}] 2022-03-08T04:43:11.600 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:43:12.370 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:43:12.371 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:43:12.188155Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:43:12.188155Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:43:12.188155Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:43:12.188155Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:43:12.188155Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:43:12.188155Z", "running": 0, "size": 1}}] 2022-03-08T04:43:22.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:43:22.287643 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:43:22.396 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:43:22.935 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:43:22.936401 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:43:22.926+0000 7fcf9e4ed380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:43:22.936 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:43:22.926+0000 7fcf9e4ed380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:43:22.936 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:43:22.926+0000 7fcf9e4ed380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:43:22.936 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:43:23.267 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:43:23.268 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:43:22.937915Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:43:22.937915Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:43:22.937915Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:43:22.937915Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:43:22.937915Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:43:22.937915Z", "running": 0, "size": 1}}] 2022-03-08T04:43:33.294 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:43:33.979 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:43:33.980 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:43:33.824323Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:43:33.824323Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:43:33.824323Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:43:33.824323Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:43:33.824323Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:43:33.824323Z", "running": 0, "size": 1}}] 2022-03-08T04:43:44.003 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:43:44.801 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:43:44.801 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:43:44.616002Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:43:44.616002Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:43:44.616002Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:43:44.616002Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:43:44.616002Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:43:44.616002Z", "running": 0, "size": 1}}] 2022-03-08T04:43:54.828 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:43:55.572 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:43:55.572 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:43:55.393216Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:43:55.393216Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:43:55.393216Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:43:55.393216Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:43:55.393216Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:43:55.393216Z", "running": 0, "size": 1}}] 2022-03-08T04:44:05.599 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:44:06.363 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:44:06.364 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:44:06.145672Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:44:06.145672Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:44:06.145672Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:44:06.145672Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:44:06.145672Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:44:06.145672Z", "running": 0, "size": 1}}] 2022-03-08T04:44:16.388 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:44:17.219 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:44:17.219 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:44:16.968953Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:44:16.968953Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:44:16.968953Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:44:16.968953Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:44:16.968953Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:44:16.968953Z", "running": 0, "size": 1}}] 2022-03-08T04:44:27.243 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:44:28.010 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:44:28.011 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:44:27.844042Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:44:27.844042Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:44:27.844042Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:44:27.844042Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:44:27.844042Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:44:27.844042Z", "running": 0, "size": 1}}] 2022-03-08T04:44:38.036 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:44:38.763 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:44:38.763 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:44:38.571493Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:44:38.571493Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:44:38.571493Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:44:38.571493Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:44:38.571493Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:44:38.571493Z", "running": 0, "size": 1}}] 2022-03-08T04:44:48.798 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:44:49.577 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:44:49.577 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:44:49.346224Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:44:49.346224Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:44:49.346224Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:44:49.346224Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:44:49.346224Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:44:49.346224Z", "running": 0, "size": 1}}] 2022-03-08T04:44:59.602 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:45:00.382 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:45:00.382 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:45:00.173774Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:45:00.173774Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:45:00.173774Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:45:00.173774Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:45:00.173774Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:45:00.173774Z", "running": 0, "size": 1}}] 2022-03-08T04:45:10.438 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:45:11.205 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:45:11.206 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:45:11.012513Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:45:11.012513Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:45:11.012513Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:45:11.012513Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:45:11.012513Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:45:11.012513Z", "running": 0, "size": 1}}] 2022-03-08T04:45:21.232 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:45:22.002 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:45:22.003 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:45:21.800081Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:45:21.800081Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:45:21.800081Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:45:21.800081Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:45:21.800081Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:45:21.800081Z", "running": 0, "size": 1}}] 2022-03-08T04:45:32.031 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:45:33.239 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:45:33.239 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:45:32.611666Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:45:32.611666Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:45:32.611666Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:45:32.611666Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:45:32.611666Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:45:32.611666Z", "running": 0, "size": 1}}] 2022-03-08T04:45:43.260 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:45:43.971 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:45:43.971 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:45:43.794935Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:45:43.794935Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:45:43.794935Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:45:43.794935Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:45:43.794935Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:45:43.794935Z", "running": 0, "size": 1}}] 2022-03-08T04:45:53.995 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:45:54.727 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:45:54.727 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:45:54.533804Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:45:54.533804Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:45:54.533804Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:45:54.533804Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:45:54.533804Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:45:54.533804Z", "running": 0, "size": 1}}] 2022-03-08T04:46:04.749 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:46:05.479 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:46:05.479 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:46:05.300977Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:46:05.300977Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:46:05.300977Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:46:05.300977Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:46:05.300977Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:46:05.300977Z", "running": 0, "size": 1}}] 2022-03-08T04:46:15.503 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:46:16.348 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:46:16.349 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:46:16.092923Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:46:16.092923Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:46:16.092923Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:46:16.092923Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:46:16.092923Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:46:16.092923Z", "running": 0, "size": 1}}] 2022-03-08T04:46:26.374 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:46:27.062 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:46:27.063 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:46:26.897067Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:46:26.897067Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:46:26.897067Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:46:26.897067Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:46:26.897067Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:46:26.897067Z", "running": 0, "size": 1}}] 2022-03-08T04:46:37.089 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:46:37.967 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:46:37.968 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:46:37.581441Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:46:37.581441Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:46:37.581441Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:46:37.581441Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:46:37.581441Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:46:37.581441Z", "running": 0, "size": 1}}] 2022-03-08T04:46:47.993 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:46:48.775 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:46:48.776 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:46:48.588338Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:46:48.588338Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:46:48.588338Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:46:48.588338Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:46:48.588338Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:46:48.588338Z", "running": 0, "size": 1}}] 2022-03-08T04:46:58.803 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:46:59.501 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:46:59.502 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:46:59.343850Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:46:59.343850Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:46:59.343850Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:46:59.343850Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:46:59.343850Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:46:59.343850Z", "running": 0, "size": 1}}] 2022-03-08T04:47:09.528 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:47:10.268 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:47:10.269 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:47:10.083446Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:47:10.083446Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:47:10.083446Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:47:10.083446Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:47:10.083446Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:47:10.083446Z", "running": 0, "size": 1}}] 2022-03-08T04:47:20.292 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:47:20.954 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:47:20.955 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:47:20.807894Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:47:20.807894Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:47:20.807894Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:47:20.807894Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:47:20.807894Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:47:20.807894Z", "running": 0, "size": 1}}] 2022-03-08T04:47:30.982 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:47:31.807 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:47:31.807 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:47:31.565428Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:47:31.565428Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:47:31.565428Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:47:31.565428Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:47:31.565428Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:47:31.565428Z", "running": 0, "size": 1}}] 2022-03-08T04:47:41.832 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:47:42.543 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:47:42.544 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:47:42.364013Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:47:42.364013Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:47:42.364013Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:47:42.364013Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:47:42.364013Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:47:42.364013Z", "running": 0, "size": 1}}] 2022-03-08T04:47:52.571 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:47:53.339 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:47:53.339 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:47:53.132069Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:47:53.132069Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:47:53.132069Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:47:53.132069Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:47:53.132069Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:47:53.132069Z", "running": 0, "size": 1}}] 2022-03-08T04:48:03.364 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-n929f -- ceph orch ls -f json 2022-03-08T04:48:04.103 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-08T04:48:04.103 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:05.000000Z", "last_refresh": "2022-03-08T04:48:03.935334Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:26:31.000000Z", "last_refresh": "2022-03-08T04:48:03.935334Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:24:06.000000Z", "last_refresh": "2022-03-08T04:48:03.935334Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-08T04:28:04.000000Z", "last_refresh": "2022-03-08T04:48:03.935334Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-08T04:48:03.935334Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-08T04:48:03.935334Z", "running": 0, "size": 1}}] 2022-03-08T04:48:04.134 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T04:48:04.135 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T04:48:04.136 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-08T04:48:04.136 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T04:48:04.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T04:48:04.171 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-08T04:48:04.190 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T04:48:04.191 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T04:48:04.206 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-08T04:48:04.223 DEBUG:teuthology.orchestra.run.smithi018:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-08T04:48:04.292 INFO:teuthology.orchestra.run.smithi018.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-08T04:48:04.304 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-08T04:48:04.305 DEBUG:teuthology.orchestra.run.smithi018:> kubectl delete -f cluster.yaml 2022-03-08T04:48:04.388 INFO:teuthology.orchestra.run.smithi018.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-08T04:48:04.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.394851 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-08T04:48:04.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.394931 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-08T04:48:04.395 INFO:tasks.rook.operator.smithi018.stdout:I0308 04:48:04.395152 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-08T04:48:04.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.395193 I | operator: successfully started the controller-runtime manager 2022-03-08T04:48:04.443 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.444493 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-08T04:48:04.444 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.444520 I | operator: watching all namespaces for Ceph CRs 2022-03-08T04:48:04.444 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.444550 I | operator: setting up schemes 2022-03-08T04:48:04.447 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.448272 I | operator: setting up the controller-runtime manager 2022-03-08T04:48:04.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.751989 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-08T04:48:04.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754296 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-08T04:48:04.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754337 I | ceph-cluster-controller: successfully started 2022-03-08T04:48:04.756 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754403 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-08T04:48:04.756 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754433 I | ceph-crashcollector-controller: successfully started 2022-03-08T04:48:04.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754462 I | ceph-block-pool-controller: successfully started 2022-03-08T04:48:04.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754505 I | ceph-object-store-user-controller: successfully started 2022-03-08T04:48:04.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754530 I | ceph-object-realm-controller: successfully started 2022-03-08T04:48:04.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754546 I | ceph-object-zonegroup-controller: successfully started 2022-03-08T04:48:04.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754561 I | ceph-object-zone-controller: successfully started 2022-03-08T04:48:04.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754710 I | ceph-object-controller: successfully started 2022-03-08T04:48:04.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754758 I | ceph-file-controller: successfully started 2022-03-08T04:48:04.759 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754799 I | ceph-nfs-controller: successfully started 2022-03-08T04:48:04.759 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754832 I | ceph-rbd-mirror-controller: successfully started 2022-03-08T04:48:04.759 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754862 I | ceph-client-controller: successfully started 2022-03-08T04:48:04.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754884 I | ceph-filesystem-mirror-controller: successfully started 2022-03-08T04:48:04.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754916 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-08T04:48:04.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754936 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-08T04:48:04.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754954 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-08T04:48:04.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754976 I | ceph-bucket-topic: successfully started 2022-03-08T04:48:04.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.754990 I | ceph-bucket-notification: successfully started 2022-03-08T04:48:04.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.755005 I | ceph-bucket-notification: successfully started 2022-03-08T04:48:04.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.755019 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-08T04:48:04.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.755975 I | operator: starting the controller-runtime manager 2022-03-08T04:48:04.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.858792 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-08T04:48:04.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.858847 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-08T04:48:04.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.858865 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-08T04:48:04.862 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.863312 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-08T04:48:04.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.904415 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:04.985 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.986178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:04.998 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:04.999599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:05.591 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:05.592596 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:05.581+0000 7fe6e3fc2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:05.592 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:05.581+0000 7fe6e3fc2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:05.592 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:05.581+0000 7fe6e3fc2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:05.592 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:05.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:05.675173 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:06.374 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:06.375606 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:06.361+0000 7f2dc3138380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:06.375 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:06.361+0000 7f2dc3138380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:06.379 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:06.361+0000 7f2dc3138380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:06.379 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:06.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:06.390652 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:07.144 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:07.145182 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:07.133+0000 7fe5c5fe2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:07.144 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:07.133+0000 7fe5c5fe2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:07.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:07.133+0000 7fe5c5fe2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:07.145 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:07.168 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:07.169951 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:07.852 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:07.853095 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:07.841+0000 7fae30693380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:07.853 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:07.841+0000 7fae30693380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:07.853 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:07.841+0000 7fae30693380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:07.854 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:07.898 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:07.899654 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:08.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:08.583873 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:08.573+0000 7fde84dbe380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:08.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:08.573+0000 7fde84dbe380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:08.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:08.573+0000 7fde84dbe380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:08.584 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:08.670 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:08.671151 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:09.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:09.302869 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:09.289+0000 7fea4076c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:09.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:09.289+0000 7fea4076c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:09.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:09.289+0000 7fea4076c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:09.303 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:09.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:09.471885 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:10.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:10.153225 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:10.141+0000 7faa396c5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:10.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:10.141+0000 7faa396c5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:10.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:10.141+0000 7faa396c5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:10.155 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:10.490 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:10.491226 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:11.158 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:11.158021 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:11.145+0000 7f9d01ae6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:11.159 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:11.145+0000 7f9d01ae6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:11.159 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:11.145+0000 7f9d01ae6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:11.159 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:11.806 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:11.807417 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:12.450 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:12.451391 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:12.437+0000 7f9505920380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:12.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:12.437+0000 7f9505920380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:12.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:12.437+0000 7f9505920380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:12.451 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:13.742 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:13.743677 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:14.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:14.389566 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:14.381+0000 7fef4d160380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:14.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:14.381+0000 7fef4d160380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:14.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:14.381+0000 7fef4d160380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:14.390 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:15.109 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:15.110334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:15.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:15.123433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:17.006 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:17.007098 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:17.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:17.687783 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:17.673+0000 7f74267e1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:17.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:17.673+0000 7f74267e1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:17.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:17.673+0000 7f74267e1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:17.687 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:22.814 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:22.815893 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:23.546 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:23.546771 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:23.533+0000 7f4b0d4af380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:23.546 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:23.533+0000 7f4b0d4af380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:23.546 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:23.533+0000 7f4b0d4af380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:23.547 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:25.201 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:25.202643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:25.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:25.246057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:33.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:33.796884 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:34.439 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:34.439566 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:34.428+0000 7ff0b59c3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:34.439 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:34.428+0000 7ff0b59c3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:34.439 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:34.428+0000 7ff0b59c3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:34.439 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:35.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:35.349583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:35.362 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:35.363909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:45.538 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:45.539262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:45.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:45.553777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:54.928 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:54.929334 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:48:55.652 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:55.652958 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:48:55.644+0000 7f2d5dac1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:48:55.652 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:55.644+0000 7f2d5dac1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:48:55.652 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:48:55.644+0000 7f2d5dac1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:48:55.653 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:48:55.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:55.662577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:48:55.675 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:48:55.675999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:05.827 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:05.828697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:05.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:05.844934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:15.935 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:15.936414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:15.944 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:15.946368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:26.005 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:26.005539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:26.015 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:26.016428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:36.087 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:36.088712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:36.097 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:36.098809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:36.664 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:36.665150 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:49:37.316 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:37.317459 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:49:37.307+0000 7f3153f9f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:49:37.317 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:49:37.307+0000 7f3153f9f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:49:37.317 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:49:37.307+0000 7f3153f9f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:49:37.317 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:49:46.192 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:46.193239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:46.204 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:46.205327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:56.269 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:56.271024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:49:56.280 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:49:56.281837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:06.365 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:06.366911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:06.376 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:06.377838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:16.438 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:16.439777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:16.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:16.452648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:26.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:26.564997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:26.578 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:26.579871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:36.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:36.673605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:36.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:36.685912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:46.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:46.870371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:46.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:46.882146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:56.963 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:56.963633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:56.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:56.976351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:50:59.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:59.245389 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:50:59.908 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:50:59.909446 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:50:59.898+0000 7f142a257380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:50:59.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:50:59.898+0000 7f142a257380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:50:59.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:50:59.898+0000 7f142a257380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:50:59.910 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:51:07.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:07.107131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:07.262 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:07.120754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:17.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:17.228038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:17.238 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:17.239616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:27.323 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:27.324442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:27.378 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:27.379288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:37.460 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:37.460475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:37.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:37.472564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:47.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:47.546680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:47.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:47.558850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:57.670 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:57.669694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:51:57.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:51:57.682115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:07.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:07.797698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:07.812 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:07.813478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:17.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:17.896256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:17.908 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:17.909938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:28.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:28.042124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:28.054 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:28.055753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:38.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:38.127324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:38.199 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:38.200349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:48.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:48.297337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:48.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:48.312713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:58.443 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:58.443795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:52:58.454 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:52:58.455858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:08.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:08.524655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:08.537 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:08.538891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:18.614 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:18.615172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:18.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:18.651869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:28.723 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:28.723359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:28.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:28.738493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:38.845 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:38.845952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:38.862 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:38.863436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:43.763 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:43.764083 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:53:44.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:44.391515 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:53:44.378+0000 7fc5d7e4c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:53:44.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:53:44.378+0000 7fc5d7e4c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:53:44.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:53:44.378+0000 7fc5d7e4c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:53:44.392 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:53:48.968 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:48.969376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:48.980 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:48.980604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:59.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:59.093859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:53:59.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:53:59.105466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:09.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:09.172798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:09.185 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:09.185661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:19.368 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:19.368437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:19.380 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:19.381177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:29.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:29.552593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:29.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:29.563188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:39.635 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:39.635520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:39.649 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:39.649903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:49.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:49.733518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:49.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:49.747313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:59.823 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:59.823287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:54:59.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:54:59.886873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:09.958 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:09.958171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:09.970 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:09.971177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:20.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:20.084936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:20.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:20.100906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:30.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:30.233692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:30.242 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:30.243115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:40.355 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:40.356125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:40.366 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:40.367520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:50.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:50.469802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:55:50.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:55:50.481522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:00.590 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:00.590961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:00.603 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:00.604060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:10.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:10.663361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:10.675 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:10.675752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:20.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:20.754369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:20.768 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:20.768279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:30.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:30.883758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:30.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:30.895905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:40.973 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:40.972964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:40.984 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:40.984727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:51.058 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:51.058108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:56:51.105 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:56:51.105841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:01.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:01.179657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:01.193 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:01.193231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:11.314 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:11.314519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:11.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:11.346882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:21.409 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:21.407023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:21.422 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:21.422613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:31.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:31.489815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:31.503 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:31.503039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:41.604 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:41.603904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:41.615 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:41.615901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:51.738 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:51.738480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:57:51.752 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:57:51.752432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:01.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:01.967024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:01.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:01.979899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:12.071 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:12.070945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:12.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:12.083726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:22.183 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:22.183096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:22.195 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:22.195367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:32.265 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:32.265021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:32.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:32.284352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:42.355 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:42.354830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:42.397 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:42.397779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:52.528 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:52.528003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:58:52.542 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:58:52.542906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:02.650 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:02.650550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:02.715 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:02.714956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:12.103 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:12.102713 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T04:59:12.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:12.797948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:12.807 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:12.805644 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T04:59:12.796+0000 7f42636be380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T04:59:12.807 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:59:12.796+0000 7f42636be380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T04:59:12.808 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T04:59:12.796+0000 7f42636be380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T04:59:12.809 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T04:59:12.810 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:12.809725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:22.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:22.902678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:22.914 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:22.914518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:32.982 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:32.981980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:32.995 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:32.995868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:43.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:43.080760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:43.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:43.096301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:53.199 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:53.199438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T04:59:53.212 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 04:59:53.212215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:03.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:03.348281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:03.359 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:03.359641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:13.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:13.469680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:13.482 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:13.482955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:23.548 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:23.547532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:23.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:23.559471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:33.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:33.617738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:33.635 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:33.635097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:43.729 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:43.727556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:43.741 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:43.741247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:53.837 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:53.836793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:00:53.850 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:00:53.850005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:03.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:03.974881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:03.984 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:03.984813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:14.139 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:14.138920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:14.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:14.154719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:24.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:24.385918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:24.398 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:24.397925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:34.463 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:34.463128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:34.522 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:34.521810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:44.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:44.629563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:44.645 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:44.645245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:54.748 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:54.748026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:01:54.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:01:54.761472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:04.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:04.876062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:04.950 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:04.949598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:15.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:15.040533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:15.052 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:15.052513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:25.166 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:25.166199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:25.178 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:25.178310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:35.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:35.285756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:35.297 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:35.296937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:45.384 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:45.383936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:45.396 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:45.395997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:55.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:55.484830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:02:55.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:02:55.499123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:05.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:05.567177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:05.581 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:05.580767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:15.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:15.661003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:15.675 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:15.675327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:25.752 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:25.752286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:25.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:25.797528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:35.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:35.870236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:35.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:35.880398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:45.971 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:45.970768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:45.983 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:45.983076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:56.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:56.076948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:03:56.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:03:56.092333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:06.200 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:06.200205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:06.214 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:06.213677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:16.320 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:16.319902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:16.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:16.334403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:26.403 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:26.402735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:26.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:26.470538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:36.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:36.622287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:36.634 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:36.634588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:46.728 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:46.728364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:46.742 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:46.742077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:56.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:56.807357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:04:56.825 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:04:56.825406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:06.944 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:06.944088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:06.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:06.954303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:17.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:17.036918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:17.052 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:17.052477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:27.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:27.121508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:27.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:27.160809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:37.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:37.284049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:37.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:37.295052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:47.412 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:47.412136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:47.424 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:47.424561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:57.525 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:57.525064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:05:57.536 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:05:57.536337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:07.638 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:07.631539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:07.645 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:07.644983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:17.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:17.769677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:17.781 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:17.781040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:27.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:27.887857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:27.901 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:27.901345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:37.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:37.990506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:38.024 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:38.023643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:48.108 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:48.107610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:48.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:48.121690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:58.221 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:58.221090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:06:58.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:06:58.235306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:08.330 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:08.330100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:08.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:08.347726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:18.423 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:18.422435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:18.437 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:18.436366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:28.735 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:28.734166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:28.748 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:28.746539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:38.807 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:38.806867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:38.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:38.816896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:48.894 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:48.893576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:48.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:48.956649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:59.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:59.037583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:07:59.047 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:07:59.047745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:09.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:09.174951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:09.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:09.187732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:19.287 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:19.287355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:19.300 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:19.296545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:29.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:29.392953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:29.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:29.405788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:39.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:39.533977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:39.548 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:39.548218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:49.647 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:49.646414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:49.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:49.658495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:59.767 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:59.766980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:08:59.779 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:08:59.778759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:09.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:09.857946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:09.868 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:09.868259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:19.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:19.942975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:19.956 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:19.956198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:30.045 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:30.044222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:30.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:30.060523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:40.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:40.157329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:40.190 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:40.189665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:50.319 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:50.318545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:09:50.331 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:09:50.330335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:00.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:00.449690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:00.461 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:00.461575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:08.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:08.174470 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T05:10:08.850 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:08.849019 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T05:10:08.839+0000 7f4ffea40380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T05:10:08.850 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T05:10:08.839+0000 7f4ffea40380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T05:10:08.850 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T05:10:08.839+0000 7f4ffea40380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T05:10:08.851 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T05:10:10.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:10.567793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:10.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:10.582880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:20.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:20.652308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:20.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:20.715457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:30.819 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:30.818507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:30.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:30.830521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:40.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:40.887655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:40.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:40.903325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:50.976 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:50.975607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:10:51.024 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:10:51.023786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:01.124 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:01.124390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:01.138 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:01.138124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:11.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:11.281389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:11.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:11.294937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:21.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:21.394534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:21.409 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:21.408955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:31.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:31.469078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:31.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:31.500992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:41.637 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:41.637071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:41.646 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:41.646896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:51.724 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:51.723826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:11:51.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:11:51.735965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:01.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:01.817364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:01.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:01.830569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:11.939 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:11.939498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:11.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:11.947719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:22.009 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:22.008821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:22.053 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:22.052382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:32.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:32.153604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:32.166 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:32.166389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:42.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:42.271262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:42.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:42.283833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:52.396 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:52.395574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:12:52.409 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:12:52.408937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:02.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:02.480015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:02.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:02.497898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:12.613 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:12.613369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:12.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:12.629596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:22.715 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:22.715306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:22.730 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:22.729786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:32.805 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:32.805331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:32.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:32.817019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:42.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:42.882383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:42.896 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:42.896451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:52.966 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:52.966190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:13:52.981 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:13:52.981057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:03.062 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:03.062200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:03.079 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:03.076279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:13.195 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:13.194940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:13.212 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:13.212292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:23.281 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:23.280521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:23.329 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:23.328159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:33.404 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:33.400356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:33.413 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:33.412933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:43.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:43.544643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:43.558 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:43.558463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:53.631 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:53.630796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:14:53.696 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:14:53.695758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:03.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:03.765327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:03.779 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:03.779100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:13.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:13.886080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:13.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:13.900095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:23.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:23.967207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:23.980 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:23.979505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:34.098 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:34.097883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:34.114 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:34.114280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:44.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:44.187055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:44.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:44.247543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:54.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:54.359838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:15:54.374 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:15:54.370307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:04.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:04.484960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:04.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:04.497523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:14.607 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:14.606295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:14.619 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:14.619444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:24.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:24.739997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:24.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:24.752836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:34.810 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:34.809550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:34.825 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:34.825112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:44.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:44.894788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:44.905 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:44.906041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:54.969 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:54.969496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:16:54.982 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:16:54.982478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:05.067 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:05.065870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:05.118 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:05.117771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:15.194 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:15.194033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:15.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:15.250494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:25.371 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:25.370502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:25.382 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:25.382198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:35.490 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:35.490320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:35.505 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:35.505527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:45.573 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:45.572689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:45.590 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:45.590209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:55.665 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:55.664561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:17:55.680 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:17:55.680505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:05.741 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:05.740848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:05.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:05.754464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:15.822 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:15.822148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:15.835 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:15.835855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:25.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:25.940556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:25.955 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:25.955392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:36.016 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:36.016554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:36.027 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:36.027771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:46.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:46.096121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:46.109 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:46.109336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:56.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:56.226262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:18:56.238 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:18:56.238443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:06.317 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:06.317046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:06.330 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:06.330731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:16.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:16.477763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:16.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:16.489410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:26.585 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:26.585013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:26.657 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:26.657018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:36.724 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:36.723781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:36.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:36.737033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:46.805 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:46.805543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:46.884 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:46.884119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:56.990 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:56.990079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:19:57.004 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:19:57.003912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:07.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:07.099903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:07.113 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:07.113525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:17.231 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:17.230743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:17.245 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:17.245214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:27.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:27.345807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:27.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:27.359960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:37.426 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:37.426081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:37.438 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:37.438079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:47.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:47.539243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:47.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:47.552913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:57.617 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:57.617291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:20:57.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:20:57.667297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:07.739 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:07.739076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:07.750 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:07.750772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:17.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:17.837664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:17.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:17.849971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:27.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:27.923009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:27.940 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:27.939802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:38.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:38.007119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:38.020 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:38.020563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:48.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:48.095084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:48.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:48.121704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:58.186 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:58.185608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:21:58.199 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:21:58.199548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:08.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:08.336388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:08.352 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:08.352122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:18.443 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:18.443642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:18.455 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:18.455664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:28.525 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:28.524742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:28.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:28.582743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:38.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:38.642525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:38.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:38.653601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:48.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:48.745994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:48.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:48.773279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:58.889 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:58.888605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:22:58.904 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:22:58.903977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:08.977 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:08.977032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:08.988 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:08.988853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:19.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:19.102492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:19.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:19.122241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:29.180 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:29.175867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:29.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:29.188249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:39.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:39.284544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:39.299 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:39.299637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:49.367 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:49.367119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:49.533 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:49.451146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:59.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:59.551682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:23:59.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:23:59.565135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:09.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:09.641571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:09.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:09.659048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:19.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:19.758356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:19.771 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:19.771795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:29.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:29.902088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:29.915 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:29.914849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:39.988 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:39.987919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:40.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:40.008640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:50.117 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:50.116636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:24:50.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:24:50.129593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:00.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:00.222703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:00.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:00.235047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:10.321 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:10.321170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:10.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:10.332458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:20.599 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:20.599190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:20.614 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:20.614753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:30.708 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:30.708447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:30.720 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:30.720863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:40.856 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:40.855973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:40.867 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:40.867286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:50.929 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:50.929037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:25:50.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:25:50.941564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:01.056 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:01.055680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:01.068 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:01.068091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:11.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:11.171605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:11.184 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:11.184085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:21.254 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:21.254061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:21.304 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:21.304362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:31.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:31.414400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:31.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:31.430310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:41.534 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:41.533784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:41.547 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:41.546976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:48.867 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:48.866750 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T05:26:49.517 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:49.516226 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T05:26:49.504+0000 7f62b931c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T05:26:49.517 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T05:26:49.504+0000 7f62b931c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T05:26:49.517 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T05:26:49.504+0000 7f62b931c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T05:26:49.517 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T05:26:51.647 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:51.647346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:26:51.657 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:26:51.657501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:01.739 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:01.739436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:01.750 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:01.750760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:11.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:11.909161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:11.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:11.923269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:22.015 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:22.015388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:22.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:22.029439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:32.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:32.093446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:32.158 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:32.158648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:42.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:42.284104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:42.297 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:42.297735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:52.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:52.360610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:27:52.373 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:27:52.373381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:02.436 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:02.436703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:02.448 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:02.448643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:12.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:12.549821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:12.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:12.564456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:22.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:22.679163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:22.690 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:22.691098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:32.825 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:32.825470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:32.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:32.838660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:42.927 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:42.927601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:42.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:42.941342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:53.068 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:53.068146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:28:53.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:28:53.082022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:03.147 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:03.147453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:03.160 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:03.161066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:13.279 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:13.278943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:13.289 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:13.290183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:23.359 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:23.359451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:23.375 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:23.375940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:33.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:33.478484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:33.490 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:33.490986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:43.610 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:43.610200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:43.641 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:43.642228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:53.752 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:53.752416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:29:53.763 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:29:53.764487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:03.857 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:03.857629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:03.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:03.870663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:13.940 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:13.941126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:13.996 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:13.955380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:24.063 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:24.063419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:24.076 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:24.077075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:34.138 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:34.138721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:34.192 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:34.192743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:44.322 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:44.323185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:44.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:44.333130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:54.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:54.407500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:30:54.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:30:54.452491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:04.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:04.553578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:04.566 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:04.566774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:14.624 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:14.624770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:14.639 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:14.639905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:24.764 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:24.764481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:24.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:24.776571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:34.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:34.881455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:34.915 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:34.910785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:45.015 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:45.015592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:45.026 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:45.027369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:55.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:55.155292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:31:55.170 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:31:55.170892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:05.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:05.235940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:05.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:05.286559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:15.421 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:15.421644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:15.434 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:15.435299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:25.551 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:25.552256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:25.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:25.565059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:35.634 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:35.635416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:35.695 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:35.696017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:45.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:45.785757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:45.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:45.800557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:55.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:55.871019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:32:55.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:32:55.881007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:05.945 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:05.946495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:05.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:05.957953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:16.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:16.042552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:16.058 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:16.059036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:26.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:26.128386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:26.140 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:26.141321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:36.231 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:36.232373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:36.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:36.244658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:46.358 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:46.358559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:46.370 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:46.371133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:56.443 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:56.443900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:33:56.455 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:33:56.455976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:06.524 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:06.525556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:06.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:06.536936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:16.606 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:16.607340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:16.620 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:16.621190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:26.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:26.685154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:26.696 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:26.697254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:36.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:36.818270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:36.828 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:36.829537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:46.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:46.889603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:46.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:46.900719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:56.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:56.976183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:34:57.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:34:57.013600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:07.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:07.082117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:07.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:07.094308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:17.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:17.238566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:17.252 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:17.253763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:27.357 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:27.358613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:27.373 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:27.374403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:37.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:37.470228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:37.483 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:37.484366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:47.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:47.589003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:47.602 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:47.603302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:57.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:57.682115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:35:57.695 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:35:57.696928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:07.811 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:07.811917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:07.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:07.825819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:17.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:17.896317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:17.907 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:17.908539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:27.995 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:27.995291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:28.006 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:28.007760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:38.128 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:38.129247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:38.141 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:38.142613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:48.207 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:48.208475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:48.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:48.218208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:58.339 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:58.339848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:36:58.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:36:58.389266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:08.496 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:08.497377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:08.511 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:08.512528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:18.608 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:18.608890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:18.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:18.623795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:28.727 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:28.727762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:28.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:28.744502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:38.847 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:38.848616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:38.862 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:38.863404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:48.945 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:48.946623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:48.959 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:48.960293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:59.058 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:59.058639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:37:59.075 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:37:59.076480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:09.156 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:09.157499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:09.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:09.172910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:19.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:19.313584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:19.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:19.327112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:29.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:29.393520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:29.448 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:29.449547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:39.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:39.791139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:39.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:39.804847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:49.963 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:49.964071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:38:49.974 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:38:49.975728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:00.068 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:00.068938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:00.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:00.100362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:10.164 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:10.165549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:10.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:10.216712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:20.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:20.296313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:20.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:20.315426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:30.429 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:30.430218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:30.440 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:30.441432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:40.507 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:40.508306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:40.520 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:40.521579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:50.593 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:50.594900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:39:50.607 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:39:50.608277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:00.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:00.680651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:00.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:00.712060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:10.780 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:10.780024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:10.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:10.795637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:20.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:20.900265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:20.912 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:20.913409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:31.023 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:31.024239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:31.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:31.038338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:41.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:41.122230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:41.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:41.132842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:51.212 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:51.213627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:40:51.225 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:40:51.226901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:01.333 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:01.335261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:01.349 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:01.347335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:11.418 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:11.420212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:11.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:11.433275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:21.524 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:21.525736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:21.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:21.540965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:31.649 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:31.650234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:31.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:31.664413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:41.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:41.747060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:41.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:41.760445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:51.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:51.870719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:41:51.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:41:51.883091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:01.983 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:01.985028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:01.997 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:01.999504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:12.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:12.107587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:12.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:12.120453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:22.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:22.189505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:22.201 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:22.202515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:32.347 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:32.348803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:32.357 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:32.358657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:42.466 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:42.468080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:42.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:42.480719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:52.613 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:52.614835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:42:52.626 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:42:52.627596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:02.781 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:02.783273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:02.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:02.796323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:12.951 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:12.952935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:12.964 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:12.965471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:23.059 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:23.061035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:23.071 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:23.072821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:29.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:29.525095 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T05:43:30.254 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:30.255465 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T05:43:30.245+0000 7fcaf25af380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T05:43:30.255 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T05:43:30.245+0000 7fcaf25af380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T05:43:30.255 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T05:43:30.245+0000 7fcaf25af380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T05:43:30.255 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T05:43:33.191 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:33.193030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:33.203 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:33.204750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:43.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:43.314470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:43.326 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:43.327988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:53.402 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:53.403694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:43:53.447 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:43:53.449015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:03.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:03.569790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:03.577 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:03.579620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:13.683 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:13.681243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:13.692 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:13.694431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:23.820 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:23.822407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:23.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:23.835703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:33.917 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:33.919495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:33.929 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:33.931530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:44.060 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:44.061939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:44.072 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:44.073913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:54.239 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:54.240629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:44:54.255 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:44:54.253920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:04.379 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:04.380551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:04.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:04.397537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:14.513 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:14.515070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:14.526 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:14.527719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:24.614 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:24.616015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:24.626 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:24.628075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:34.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:34.711868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:34.721 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:34.722884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:44.786 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:44.787668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:44.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:44.800382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:54.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:54.883062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:45:54.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:45:54.897669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:04.992 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:04.993789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:05.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:05.005347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:15.114 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:15.115896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:15.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:15.128874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:25.195 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:25.197379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:25.212 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:25.214221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:35.362 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:35.363650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:35.373 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:35.375783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:45.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:45.489069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:45.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:45.503544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:55.570 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:55.571899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:46:55.581 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:46:55.583658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:05.673 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:05.675320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:05.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:05.687938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:15.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:15.744484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:15.756 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:15.757940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:25.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:25.871645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:25.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:25.882402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:35.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:35.942697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:36.009 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:36.011191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:46.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:46.106151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:46.116 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:46.118297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:56.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:56.219426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:47:56.234 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:47:56.236629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:06.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:06.353257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:06.361 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:06.363457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:16.435 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:16.436215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:16.445 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:16.447279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:26.526 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:26.527008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:26.538 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:26.540399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:36.673 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:36.674983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:36.690 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:36.687797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:46.808 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:46.809088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:46.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:46.818772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:56.887 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:56.889353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:48:56.901 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:48:56.903315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:07.026 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:07.027672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:07.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:07.040226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:17.112 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:17.114469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:17.125 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:17.127476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:27.230 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:27.231637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:27.245 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:27.247525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:37.349 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:37.351006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:37.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:37.362505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:47.429 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:47.431391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:47.444 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:47.446152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:57.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:57.565202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:49:57.576 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:49:57.578725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:07.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:07.686849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:07.701 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:07.702957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:17.823 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:17.824843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:17.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:17.835321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:27.912 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:27.914084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:27.970 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:27.972257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:38.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:38.096366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:38.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:38.108581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:48.240 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:48.241472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:48.249 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:48.251558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:58.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:58.391043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:50:58.401 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:50:58.403377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:08.494 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:08.496321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:08.507 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:08.509329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:18.636 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:18.638090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:18.648 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:18.649916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:28.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:28.719762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:28.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:28.735003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:38.842 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:38.843850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:38.852 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:38.854042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:48.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:48.947603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:48.958 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:48.960487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:59.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:59.063040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:51:59.075 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:51:59.076701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:09.135 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:09.137269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:09.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:09.148881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:19.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:19.250071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:19.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:19.265332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:29.337 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:29.337245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:29.345 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:29.347220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:39.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:39.470223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:39.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:39.483290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:49.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:49.563998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:49.609 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:49.611431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:59.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:59.686100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:52:59.696 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:52:59.698301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:09.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:09.770952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:09.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:09.835038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:19.910 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:19.911689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:19.974 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:19.976258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:30.072 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:30.073776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:30.085 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:30.086979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:40.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:40.152960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:40.166 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:40.168065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:50.270 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:50.272352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:50.281 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:53:50.283230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:00.353 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:00.355742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:00.366 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:00.368591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:10.515 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:10.516658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:10.526 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:10.528521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:20.602 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:20.604451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:20.617 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:20.619575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:30.723 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:30.724991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:30.734 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:30.736726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:40.813 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:40.815018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:40.864 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:40.866627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:50.985 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:50.986762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:54:50.998 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:54:51.000372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:01.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:01.108899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:01.170 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:01.172602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:11.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:11.250337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:11.261 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:11.263897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:21.378 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:21.380269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:21.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:21.394729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:31.503 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:31.504580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:31.516 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:31.518123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:41.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:41.652476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:41.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:41.665104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:51.767 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:51.768512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:55:51.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:55:51.780907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:01.861 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:01.863200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:01.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:01.874639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:11.989 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:11.991244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:12.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:12.005488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:22.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:22.108769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:22.139 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:22.141408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:32.202 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:32.203238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:32.216 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:32.217932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:42.324 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:42.326341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:42.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:42.353650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:52.423 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:52.424901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:56:52.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:56:52.499389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:02.594 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:02.596076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:02.602 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:02.604966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:12.702 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:12.704369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:12.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:12.717772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:22.816 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:22.818294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:22.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:22.830968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:32.907 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:32.909317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:32.920 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:32.922030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:42.987 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:42.988995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:43.047 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:43.047219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:53.109 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:53.110786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:57:53.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:57:53.120936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:03.239 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:03.239787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:03.249 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:03.251493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:13.369 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:13.370440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:13.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:13.389641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:23.498 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:23.500102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:23.510 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:23.512952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:33.611 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:33.612368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:33.620 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:33.622581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:43.692 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:43.693736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:43.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:43.747288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:53.818 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:53.820042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:58:53.868 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:58:53.870036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:03.971 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:03.972788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:03.985 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:03.987710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:14.109 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:14.110592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:14.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:14.124128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:24.224 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:24.226611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:24.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:24.239590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:34.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:34.313959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:34.363 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:34.365023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:44.475 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:44.476658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:44.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:44.489608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:54.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:54.590364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:59:54.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 05:59:54.624492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:04.711 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:04.713319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:04.723 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:04.725752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:10.263 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:10.265389 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T06:00:10.926 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:10.927951 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T06:00:10.918+0000 7f4317c66380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T06:00:10.927 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T06:00:10.918+0000 7f4317c66380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T06:00:10.927 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T06:00:10.918+0000 7f4317c66380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T06:00:10.927 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T06:00:14.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:14.834559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:14.844 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:14.846341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:24.904 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:24.906544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:24.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:24.976577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:35.091 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:35.093636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:35.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:35.106798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:45.206 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:45.207547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:45.218 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:45.220256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:55.338 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:55.340292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:00:55.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:00:55.353917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:05.460 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:05.461711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:05.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:05.473650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:15.537 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:15.539436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:15.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:15.551551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:25.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:25.660883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:25.670 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:25.672527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:35.739 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:35.741521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:35.752 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:35.754525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:45.865 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:45.866875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:45.878 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:45.880567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:55.936 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:55.934789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:01:55.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:01:55.943184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:06.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:06.030369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:06.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:06.042963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:16.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:16.150603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:16.162 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:16.163889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:26.298 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:26.299321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:26.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:26.311893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:36.460 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:36.461545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:36.473 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:36.475113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:46.550 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:46.551965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:46.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:46.565583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:56.650 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:56.650424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:02:56.662 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:02:56.662944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:06.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:06.734485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:06.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:06.749565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:16.846 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:16.847974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:16.856 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:16.858159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:26.981 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:26.982696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:26.995 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:26.996813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:37.105 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:37.106364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:37.118 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:37.120244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:47.224 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:47.225293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:47.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:47.237313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:57.341 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:57.342422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:03:57.356 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:03:57.356872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:07.458 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:07.459060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:07.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:07.470873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:17.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:17.540260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:17.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:17.554165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:27.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:27.659761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:27.673 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:27.673547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:37.749 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:37.750670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:37.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:37.762258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:47.863 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:47.864559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:47.875 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:47.876788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:57.963 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:57.964304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:04:58.013 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:04:58.014737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:08.082 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:08.083758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:08.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:08.107547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:18.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:18.389306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:18.401 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:18.402546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:28.516 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:28.516831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:28.531 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:28.529790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:38.728 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:38.728619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:38.744 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:38.745128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:48.998 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:48.998798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:49.010 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:49.011376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:59.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:59.129790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:05:59.140 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:05:59.141475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:09.201 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:09.201691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:09.212 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:09.213811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:19.318 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:19.319016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:19.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:19.332801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:29.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:29.406098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:29.424 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:29.418444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:39.518 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:39.518856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:39.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:39.535945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:49.638 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:49.638690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:49.650 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:49.650776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:59.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:59.719647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:06:59.739 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:06:59.736229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:09.813 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:09.813499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:09.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:09.831583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:19.942 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:19.942802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:19.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:19.957697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:30.069 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:30.066368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:30.080 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:30.079139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:40.140 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:40.141248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:40.158 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:40.159175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:50.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:50.259369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:07:50.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:07:50.269157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:00.361 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:00.362087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:00.374 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:00.375451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:10.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:10.468067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:10.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:10.480043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:20.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:20.583453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:20.597 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:20.597656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:30.671 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:30.669860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:30.688 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:30.682511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:40.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:40.774742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:40.788 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:40.788831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:50.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:50.869413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:08:50.921 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:08:50.921768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:01.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:01.036196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:01.049 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:01.049561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:11.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:11.126762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:11.139 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:11.139964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:21.211 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:21.211912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:21.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:21.227012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:31.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:31.296149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:31.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:31.311668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:41.444 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:41.444814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:41.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:41.456842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:51.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:51.534909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:09:51.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:09:51.549497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:01.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:01.618770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:01.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:01.630586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:11.702 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:11.702790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:11.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:11.751155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:21.854 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:21.854523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:21.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:21.870308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:31.958 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:31.958991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:31.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:31.968232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:42.055 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:42.054919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:42.067 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:42.067684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:52.128 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:52.128316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:10:52.138 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:10:52.138964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:02.262 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:02.262488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:02.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:02.275908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:12.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:12.351519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:12.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:12.364384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:22.475 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:22.475396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:22.486 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:22.487000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:32.609 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:32.609284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:32.620 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:32.620876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:42.699 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:42.699050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:42.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:42.709622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:52.784 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:52.783890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:11:52.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:11:52.802305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:02.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:02.893010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:02.908 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:02.908298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:13.008 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:13.008516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:13.020 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:13.020465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:23.088 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:23.088139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:23.128 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:23.128414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:33.352 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:33.352363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:33.366 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:33.365986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:43.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:43.477311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:43.492 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:43.492534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:53.586 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:53.586100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:12:53.600 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:12:53.600206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:03.726 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:03.725777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:03.738 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:03.738858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:13.884 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:13.884075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:13.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:13.895654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:23.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:23.954385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:23.968 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:23.968508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:34.049 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:34.048663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:34.067 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:34.066653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:44.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:44.155401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:44.167 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:44.166781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:54.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:54.282697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:13:54.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:13:54.310160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:04.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:04.415267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:04.427 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:04.427124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:14.538 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:14.538322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:14.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:14.552714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:24.647 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:24.647233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:24.662 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:24.662509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:34.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:34.777766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:34.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:34.795187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:44.916 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:44.915252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:44.931 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:44.930682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:55.070 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:55.069964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:14:55.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:14:55.083450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:05.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:05.153353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:05.167 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:05.167460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:15.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:15.248451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:15.261 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:15.261236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:25.363 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:25.362541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:25.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:25.392840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:35.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:35.479834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:35.492 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:35.492710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:45.581 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:45.580987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:45.655 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:45.655122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:55.806 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:55.806384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:15:55.816 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:15:55.816244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:05.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:05.923247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:05.936 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:05.936638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:16.013 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:16.012612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:16.022 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:16.022282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:26.087 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:26.086829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:26.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:26.100602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:36.165 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:36.164364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:36.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:36.223304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:46.329 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:46.328870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:46.341 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:46.340869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:50.933 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:50.933513 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T06:16:51.586 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:51.585463 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T06:16:51.575+0000 7f800e0c2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T06:16:51.586 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T06:16:51.575+0000 7f800e0c2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T06:16:51.587 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T06:16:51.575+0000 7f800e0c2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T06:16:51.587 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T06:16:56.423 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:56.422723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:16:56.433 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:16:56.433667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:06.491 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:06.487093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:06.540 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:06.539584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:16.670 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:16.669991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:16.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:16.681457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:26.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:26.746182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:26.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:26.757955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:36.831 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:36.831185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:36.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:36.842431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:46.942 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:46.941967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:46.956 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:46.956119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:57.028 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:57.027418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:17:57.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:17:57.060838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:07.164 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:07.164075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:07.177 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:07.177310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:17.287 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:17.286788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:17.300 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:17.300334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:27.404 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:27.403703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:27.416 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:27.416456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:37.556 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:37.555167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:37.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:37.569153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:47.676 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:47.675139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:47.695 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:47.691085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:57.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:57.757581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:18:57.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:18:57.772229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:07.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:07.894668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:07.912 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:07.911178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:18.111 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:18.109522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:18.124 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:18.124381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:28.224 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:28.223769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:28.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:28.237031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:38.326 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:38.325858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:38.338 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:38.338353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:48.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:48.413115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:48.493 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:48.492651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:58.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:58.582271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:19:58.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:19:58.629940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:08.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:08.746026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:08.762 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:08.759630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:18.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:18.828526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:18.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:18.880038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:28.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:28.960054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:28.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:28.974618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:39.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:39.047795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:39.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:39.101567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:49.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:49.176885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:49.190 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:49.190186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:59.279 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:59.278901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:20:59.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:20:59.290786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:09.362 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:09.361961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:09.376 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:09.375647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:19.475 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:19.474459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:19.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:19.487462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:29.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:29.563982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:29.577 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:29.577333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:39.680 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:39.679741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:39.694 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:39.693563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:49.801 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:49.800876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:49.815 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:49.814580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:59.926 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:59.925332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:21:59.939 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:21:59.938753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:10.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:10.002907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:10.016 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:10.016288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:20.088 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:20.087291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:20.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:20.128157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:30.238 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:30.237305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:30.252 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:30.251925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:40.321 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:40.321206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:40.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:40.333829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:50.447 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:50.446407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:22:50.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:22:50.460570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:00.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:00.561018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:00.576 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:00.576104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:10.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:10.662374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:10.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:10.680993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:20.749 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:20.748662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:20.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:20.761142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:30.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:30.870104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:30.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:30.879640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:40.990 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:40.989881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:41.001 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:41.001396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:51.197 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:51.196593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:23:51.210 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:23:51.210107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:01.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:01.311899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:01.326 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:01.325533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:11.427 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:11.426606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:11.438 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:11.438129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:21.540 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:21.538965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:21.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:21.559292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:31.654 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:31.653937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:31.665 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:31.665483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:41.720 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:41.720028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:41.730 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:41.730204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:51.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:51.798215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:24:51.848 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:24:51.848020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:01.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:01.922839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:01.936 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:01.936149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:12.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:12.047187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:12.059 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:12.058799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:22.167 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:22.166446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:22.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:22.178819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:32.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:32.285711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:32.298 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:32.298000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:42.402 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:42.401662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:42.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:42.413691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:52.513 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:52.512233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:25:52.525 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:25:52.525321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:02.657 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:02.656846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:02.668 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:02.668365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:12.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:12.790002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:12.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:12.803386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:22.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:22.882208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:22.894 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:22.894398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:32.984 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:32.982710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:32.998 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:32.997556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:43.069 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:43.068349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:43.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:43.082391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:53.170 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:53.169689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:26:53.184 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:26:53.184196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:03.309 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:03.308977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:03.320 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:03.320213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:13.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:13.387910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:13.464 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:13.463307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:23.616 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:23.615222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:23.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:23.626983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:33.725 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:33.724556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:33.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:33.735398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:43.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:43.832720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:43.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:43.843155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:53.966 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:53.965598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:27:53.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:27:53.978368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:04.056 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:04.055509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:04.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:04.101145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:14.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:14.204785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:14.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:14.216402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:24.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:24.347886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:24.358 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:24.357573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:34.425 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:34.424941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:34.436 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:34.436384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:44.541 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:44.540465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:44.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:44.556162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:54.619 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:54.617946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:28:54.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:28:54.629206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:04.725 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:04.724837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:04.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:04.736339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:14.812 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:14.811690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:14.823 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:14.823176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:24.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:24.892731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:24.931 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:24.931228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:35.005 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:35.004512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:35.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:35.018716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:45.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:45.132022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:45.147 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:45.146951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:55.264 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:55.263592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:29:55.277 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:29:55.276835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:05.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:05.350442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:05.365 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:05.365216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:15.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:15.464784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:15.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:15.479168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:25.578 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:25.576844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:25.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:25.588889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:35.708 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:35.706862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:35.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:35.719091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:45.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:45.822842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:45.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:45.838207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:55.931 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:55.930264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:30:55.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:30:55.942015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:06.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:06.038140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:06.054 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:06.053537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:16.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:16.127929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:16.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:16.144817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:26.212 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:26.211143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:26.224 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:26.224129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:36.330 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:36.329061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:36.344 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:36.343721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:46.417 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:46.416185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:46.432 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:46.431286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:56.536 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:56.535359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:31:56.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:31:56.551621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:06.676 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:06.675163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:06.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:06.686713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:16.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:16.829797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:16.840 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:16.839738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:26.952 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:26.951669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:26.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:26.965412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:37.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:37.035831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:37.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:37.049555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:47.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:47.171117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:47.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:47.186670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:57.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:57.289611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:32:57.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:32:57.301050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:07.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:07.362851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:07.420 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:07.419851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:17.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:17.488892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:17.509 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:17.501204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:27.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:27.577951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:27.592 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:27.591210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:31.594 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:31.592963 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T06:33:32.308 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:32.307026 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T06:33:32.295+0000 7f5fdef94380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T06:33:32.309 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T06:33:32.295+0000 7f5fdef94380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T06:33:32.309 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T06:33:32.295+0000 7f5fdef94380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T06:33:32.309 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T06:33:37.662 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:37.661650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:37.675 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:37.674799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:47.781 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:47.780289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:47.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:47.793419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:57.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:57.871376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:33:57.884 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:33:57.883324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:08.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:08.006462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:08.025 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:08.024106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:18.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:18.094186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:18.159 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:18.158654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:28.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:28.275711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:28.308 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:28.307942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:38.445 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:38.444106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:38.453 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:38.453055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:48.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:48.621418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:48.635 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:48.635043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:58.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:58.732019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:34:58.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:34:58.744898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:08.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:08.823595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:08.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:08.837330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:18.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:18.946257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:18.960 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:18.959998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:29.079 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:29.078403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:29.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:29.092722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:39.148 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:39.147620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:39.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:39.160892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:49.224 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:49.223569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:49.265 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:49.264743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:59.383 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:59.382401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:35:59.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:35:59.394219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:09.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:09.477613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:09.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:09.534502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:19.636 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:19.635407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:19.649 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:19.649198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:29.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:29.741869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:29.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:29.754431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:39.828 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:39.827353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:39.840 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:39.839863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:49.932 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:49.930842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:36:49.945 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:36:49.944169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:00.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:00.030726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:00.043 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:00.043040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:10.123 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:10.122825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:10.182 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:10.181104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:20.280 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:20.279628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:20.292 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:20.291710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:30.402 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:30.401135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:30.412 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:30.411584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:40.520 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:40.519326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:40.532 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:40.532247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:50.652 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:50.651081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:37:50.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:37:50.662535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:00.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:00.741822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:00.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:00.754091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:10.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:10.850796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:10.867 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:10.866175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:20.927 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:20.926781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:20.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:20.940209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:31.057 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:31.056594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:31.073 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:31.072425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:41.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:41.160291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:41.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:41.170615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:51.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:51.267526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:38:51.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:38:51.281293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:01.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:01.394607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:01.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:01.406687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:11.689 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:11.688560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:11.701 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:11.700898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:21.791 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:21.790595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:21.807 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:21.806342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:31.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:31.946879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:31.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:31.960938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:42.074 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:42.073328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:42.087 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:42.086112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:52.162 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:52.161325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:39:52.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:39:52.222105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:02.368 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:02.367764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:02.381 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:02.381068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:12.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:12.478648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:12.528 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:12.527779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:22.650 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:22.649591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:22.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:22.662565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:32.822 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:32.820937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:32.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:32.833304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:42.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:42.908306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:42.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:42.945670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:53.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:53.011912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:40:53.025 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:40:53.024577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:03.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:03.095794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:03.109 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:03.108964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:13.178 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:13.177724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:13.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:13.225354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:23.379 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:23.378719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:23.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:23.393571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:33.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:33.484407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:33.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:33.498071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:43.605 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:43.604501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:43.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:43.617634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:53.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:53.685901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:41:53.698 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:41:53.697667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:03.813 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:03.812565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:03.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:03.823191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:13.898 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:13.897656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:13.907 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:13.906798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:23.978 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:23.977683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:23.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:23.989909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:34.063 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:34.062491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:34.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:34.118025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:44.229 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:44.228206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:44.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:44.243295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:54.319 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:54.318878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:42:54.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:42:54.333679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:04.397 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:04.396620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:04.473 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:04.472160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:14.533 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:14.532492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:14.590 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:14.589130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:24.688 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:24.687160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:24.708 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:24.707468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:34.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:34.833478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:34.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:34.843290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:44.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:44.940431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:44.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:44.954066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:55.067 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:55.066343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:43:55.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:43:55.080120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:05.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:05.152632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:05.299 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:05.166182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:15.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:15.234218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:15.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:15.247898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:25.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:25.335543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:25.349 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:25.348142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:35.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:35.468224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:35.482 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:35.481586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:45.594 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:45.593375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:45.606 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:45.605490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:55.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:55.667123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:44:55.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:44:55.680508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:05.777 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:05.775651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:05.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:05.794346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:15.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:15.892379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:15.906 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:15.905761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:26.013 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:26.012112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:26.025 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:26.025131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:36.134 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:36.133120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:36.147 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:36.146194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:46.220 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:46.219103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:46.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:46.270610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:56.401 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:56.401066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:45:56.411 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:45:56.410766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:06.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:06.495990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:06.513 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:06.512784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:16.581 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:16.580728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:16.593 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:16.592811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:26.666 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:26.665650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:26.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:26.677859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:36.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:36.744227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:36.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:36.793967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:46.914 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:46.913549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:46.927 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:46.926801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:57.000 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:56.999339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:46:57.011 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:46:57.010579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:07.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:07.082792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:07.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:07.098709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:17.207 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:17.205894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:17.218 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:17.217620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:27.288 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:27.287054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:27.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:27.302403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:37.376 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:37.375296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:37.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:37.391354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:47.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:47.485955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:47.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:47.500581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:57.572 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:57.571186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:47:57.584 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:47:57.583212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:07.647 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:07.646425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:07.696 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:07.695795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:17.812 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:17.811135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:17.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:17.823395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:27.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:27.956454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:27.968 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:27.968182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:38.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:38.150019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:38.200 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:38.199523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:48.307 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:48.306923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:48.320 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:48.319632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:58.408 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:58.407575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:48:58.432 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:48:58.431235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:08.519 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:08.518614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:08.530 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:08.529995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:18.611 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:18.610595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:18.624 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:18.623355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:28.735 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:28.734046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:28.748 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:28.747998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:38.823 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:38.822256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:38.837 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:38.836190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:48.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:48.955732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:48.969 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:48.967992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:59.105 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:59.104013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:49:59.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:49:59.118370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:09.213 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:09.211777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:09.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:09.226396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:12.316 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:12.315436 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T06:50:12.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:12.952886 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T06:50:12.940+0000 7fdfbde27380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T06:50:12.955 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T06:50:12.940+0000 7fdfbde27380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T06:50:12.955 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T06:50:12.940+0000 7fdfbde27380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T06:50:12.955 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T06:50:19.298 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:19.297452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:19.349 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:19.348357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:29.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:29.413841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:29.490 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:29.489303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:39.624 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:39.623350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:39.640 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:39.638984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:49.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:49.752204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:49.767 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:49.766130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:50:59.916 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:59.914637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:00.060 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:50:59.928192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:10.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:10.011317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:10.024 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:10.023654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:20.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:20.152589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:20.167 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:20.166680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:30.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:30.223791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:30.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:30.236857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:40.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:40.334343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:40.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:40.350272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:50.424 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:50.422796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:51:50.435 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:51:50.435207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:00.504 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:00.503986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:00.517 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:00.516962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:10.587 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:10.586843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:10.601 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:10.600637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:20.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:20.668982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:20.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:20.681656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:30.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:30.857097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:30.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:30.868831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:40.956 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:40.955167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:40.973 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:40.972862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:51.076 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:51.075236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:52:51.090 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:52:51.090157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:01.231 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:01.229843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:01.242 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:01.241568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:11.361 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:11.360490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:11.373 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:11.372253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:21.531 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:21.529767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:21.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:21.542295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:31.646 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:31.645317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:31.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:31.659033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:41.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:41.768385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:41.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:41.777977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:51.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:51.871213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:53:51.884 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:53:51.883531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:01.989 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:01.988012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:02.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:02.002336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:12.109 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:12.107802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:12.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:12.119290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:22.185 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:22.184380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:22.198 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:22.197537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:32.304 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:32.303426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:32.319 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:32.319117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:42.433 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:42.432036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:42.445 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:42.444704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:52.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:52.556374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:54:52.573 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:54:52.572126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:02.650 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:02.649026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:02.664 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:02.663273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:12.777 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:12.776592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:12.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:12.789940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:22.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:22.894617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:22.910 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:22.909815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:33.021 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:33.020882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:33.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:33.033460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:43.143 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:43.142310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:43.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:43.157004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:53.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:53.270638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:55:53.283 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:55:53.282961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:03.397 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:03.396830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:03.412 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:03.412163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:13.484 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:13.483656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:13.498 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:13.497558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:23.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:23.568629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:23.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:23.626188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:33.709 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:33.707960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:33.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:33.768230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:43.855 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:43.854040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:43.867 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:43.866692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:53.934 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:53.932990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:56:53.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:56:53.946050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:04.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:04.033196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:04.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:04.047450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:14.143 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:14.142117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:14.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:14.153920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:24.263 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:24.262185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:24.275 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:24.274526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:34.345 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:34.344278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:34.358 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:34.357488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:44.458 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:44.456950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:44.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:44.469853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:54.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:54.561814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:57:54.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:57:54.578485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:04.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:04.686475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:04.698 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:04.697773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:14.767 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:14.765916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:14.827 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:14.825627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:24.887 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:24.886632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:24.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:24.901335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:35.020 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:35.019620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:35.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:35.037061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:45.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:45.118865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:45.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:45.130862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:55.239 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:55.237112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:58:55.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:58:55.249538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:05.339 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:05.338621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:05.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:05.391116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:15.461 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:15.460536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:15.473 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:15.472959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:25.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:25.561604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:25.576 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:25.575718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:35.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:35.789336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:35.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:35.801581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:45.867 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:45.866888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:45.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:45.879167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:55.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:55.940447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T06:59:55.953 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 06:59:55.952866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:06.053 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:06.052129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:06.069 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:06.068830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:16.182 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:16.180790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:16.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:16.214558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:26.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:26.333152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:26.345 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:26.344377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:36.411 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:36.409446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:36.425 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:36.424217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:46.496 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:46.493172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:46.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:46.506108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:56.657 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:56.655932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:00:56.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:00:56.668360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:06.747 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:06.746042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:06.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:06.759250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:16.875 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:16.874758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:16.889 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:16.888893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:26.987 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:26.986941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:27.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:27.002495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:37.073 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:37.072107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:37.085 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:37.084989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:47.191 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:47.190706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:47.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:47.204236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:57.344 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:57.343389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:01:57.356 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:01:57.356219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:07.416 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:07.415552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:07.444 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:07.443736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:17.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:17.496999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:17.558 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:17.557552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:27.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:27.658232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:27.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:27.680824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:37.932 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:37.931639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:37.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:37.946376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:48.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:48.032337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:48.047 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:48.046279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:58.116 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:58.114858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:02:58.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:02:58.128555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:08.200 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:08.196368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:08.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:08.216704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:18.337 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:18.336712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:18.349 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:18.348989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:28.419 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:28.418468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:28.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:28.430410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:38.621 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:38.620308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:38.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:38.629646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:48.706 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:48.704180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:48.718 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:48.717232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:58.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:58.828365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:03:58.841 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:03:58.840746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:08.908 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:08.906927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:08.920 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:08.919753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:18.988 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:18.986711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:19.000 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:18.999682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:29.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:29.101041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:29.114 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:29.113278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:39.220 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:39.219336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:39.234 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:39.233535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:49.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:49.301154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:49.363 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:49.361611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:59.461 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:59.459375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:04:59.474 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:04:59.472392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:09.574 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:09.573797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:09.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:09.589030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:19.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:19.694631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:19.707 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:19.706781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:29.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:29.798586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:29.813 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:29.812472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:39.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:39.887908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:39.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:39.956245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:50.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:50.031652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:05:50.056 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:05:50.045708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:00.110 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:00.109459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:00.125 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:00.121044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:10.198 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:10.194529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:10.207 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:10.206645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:20.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:20.295097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:20.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:20.310600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:30.375 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:30.373646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:30.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:30.429472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:40.518 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:40.516134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:40.531 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:40.530010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:50.604 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:50.602997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:50.615 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:50.614270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:06:52.968 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:52.967019 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T07:06:53.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:06:53.627860 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T07:06:53.613+0000 7f1d9c24e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T07:06:53.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T07:06:53.613+0000 7f1d9c24e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T07:06:53.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T07:06:53.613+0000 7f1d9c24e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T07:06:53.630 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T07:07:00.727 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:00.719743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:00.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:00.733081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:10.864 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:10.859557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:10.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:10.880231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:20.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:20.944914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:20.960 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:20.959940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:31.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:31.039464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:31.051 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:31.051119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:41.134 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:41.132404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:41.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:41.145804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:51.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:51.267268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:07:51.283 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:07:51.282644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:01.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:01.399489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:01.412 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:01.411852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:11.520 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:11.519443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:11.534 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:11.533090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:21.603 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:21.601986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:21.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:21.660966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:31.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:31.782729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:31.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:31.793341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:41.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:41.922808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:41.936 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:41.935757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:52.009 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:52.006773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:08:52.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:08:52.019191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:02.116 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:02.115461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:02.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:02.128763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:12.232 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:12.231388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:12.245 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:12.244299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:22.372 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:22.370702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:22.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:22.389717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:32.466 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:32.464193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:32.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:32.511394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:42.613 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:42.612077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:42.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:42.626728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:52.706 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:52.704954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:09:52.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:09:52.757104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:02.831 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:02.829243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:02.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:02.842383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:12.955 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:12.954363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:12.970 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:12.969665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:23.076 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:23.074894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:23.091 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:23.087681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:33.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:33.183553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:33.197 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:33.196626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:43.272 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:43.270416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:43.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:43.334267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:53.428 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:53.426785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:10:53.441 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:10:53.440301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:03.550 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:03.549478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:03.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:03.564022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:13.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:13.631042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:13.688 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:13.687446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:23.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:23.758047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:23.771 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:23.770892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:33.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:33.868311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:33.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:33.880163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:43.989 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:43.988493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:44.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:44.000639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:54.165 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:54.163167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:11:54.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:11:54.178422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:04.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:04.290538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:04.300 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:04.300409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:14.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:14.414364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:14.428 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:14.427475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:24.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:24.511737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:24.527 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:24.526255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:34.593 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:34.592508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:34.634 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:34.633254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:44.703 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:44.702744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:44.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:44.715633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:54.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:54.801226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:12:54.810 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:12:54.809790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:04.885 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:04.881203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:04.935 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:04.934446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:15.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:15.040285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:15.053 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:15.052837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:25.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:25.148356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:25.160 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:25.159916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:35.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:35.226338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:35.245 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:35.240089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:45.341 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:45.340537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:45.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:45.351306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:55.449 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:55.447591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:13:55.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:13:55.463997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:05.525 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:05.523592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:05.538 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:05.535215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:15.634 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:15.633265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:15.645 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:15.644850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:25.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:25.715109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:25.728 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:25.728043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:35.839 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:35.837962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:35.855 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:35.854116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:45.920 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:45.920202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:45.933 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:45.932726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:56.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:56.002557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:14:56.017 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:14:56.016641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:06.111 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:06.110742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:06.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:06.121925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:16.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:16.186070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:16.199 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:16.199250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:26.319 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:26.318544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:26.333 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:26.331863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:36.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:36.450535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:36.463 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:36.462611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:46.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:46.582451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:46.597 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:46.596483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:56.822 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:56.818859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:15:56.927 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:15:56.927290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:07.062 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:07.061123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:07.076 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:07.074230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:17.147 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:17.146232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:17.159 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:17.158661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:27.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:27.234257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:27.249 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:27.248998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:37.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:37.300689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:37.329 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:37.314534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:47.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:47.384538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:47.409 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:47.400876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:57.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:57.470536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:16:57.521 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:16:57.519841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:07.590 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:07.589891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:07.603 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:07.602764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:17.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:17.680054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:17.694 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:17.694205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:27.808 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:27.807337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:27.823 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:27.821945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:37.929 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:37.928756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:37.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:37.942942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:48.015 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:48.014934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:48.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:48.028517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:58.103 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:58.102139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:17:58.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:17:58.160353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:08.239 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:08.239096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:08.315 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:08.293039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:18.403 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:18.402170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:18.423 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:18.416432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:28.494 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:28.493711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:28.506 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:28.505449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:38.575 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:38.574624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:38.586 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:38.585835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:48.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:48.665952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:48.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:48.678524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:58.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:58.799558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:18:58.816 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:18:58.810707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:08.972 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:08.971899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:08.985 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:08.984263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:19.110 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:19.106757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:19.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:19.118678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:29.323 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:29.321938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:29.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:29.335124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:39.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:39.496136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:39.506 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:39.506107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:49.625 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:49.623921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:49.636 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:49.636123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:59.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:59.777290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:19:59.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:19:59.789749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:09.911 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:09.909519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:09.922 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:09.921189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:20.030 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:20.029136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:20.045 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:20.044642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:30.143 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:30.142168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:30.189 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:30.189002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:40.329 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:40.328231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:40.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:40.344341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:50.412 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:50.411286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:20:50.423 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:20:50.423281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:00.492 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:00.487247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:00.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:00.501307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:10.656 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:10.655428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:10.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:10.666981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:20.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:20.752718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:20.766 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:20.765735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:30.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:30.832063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:30.844 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:30.844073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:40.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:40.961071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:40.977 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:40.974798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:51.075 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:51.075067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:21:51.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:21:51.092041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:01.196 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:01.195906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:01.208 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:01.207708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:11.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:11.285776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:11.301 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:11.300771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:21.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:21.385535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:21.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:21.399994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:31.509 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:31.508542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:31.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:31.522246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:41.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:41.629543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:41.643 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:41.642650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:51.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:51.759224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:22:51.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:22:51.771741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:01.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:01.850481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:01.864 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:01.863875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:11.939 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:11.938167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:11.951 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:11.949864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:22.016 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:22.014079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:22.028 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:22.027598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:32.123 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:32.122168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:32.136 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:32.135767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:33.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:33.638752 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T07:23:34.363 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:34.358631 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T07:23:34.346+0000 7f5da14ce380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T07:23:34.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T07:23:34.346+0000 7f5da14ce380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T07:23:34.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T07:23:34.346+0000 7f5da14ce380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T07:23:34.364 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T07:23:42.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:42.255568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:42.270 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:42.270054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:52.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:52.335509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:23:52.382 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:23:52.382121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:02.502 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:02.501876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:02.524 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:02.523844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:12.638 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:12.636098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:12.649 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:12.649254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:22.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:22.770038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:22.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:22.784906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:32.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:32.891246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:32.905 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:32.904799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:43.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:43.040884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:43.053 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:43.052867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:53.148 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:53.147928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:24:53.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:24:53.161007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:03.265 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:03.264300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:03.283 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:03.283026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:13.350 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:13.349409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:13.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:13.406222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:23.483 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:23.482724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:23.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:23.538250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:33.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:33.681010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:33.694 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:33.693570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:43.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:43.761280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:43.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:43.817033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:53.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:53.881569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:25:53.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:25:53.895185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:03.977 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:03.976362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:03.989 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:03.988948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:14.052 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:14.051897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:14.068 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:14.067585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:24.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:24.168279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:24.185 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:24.181609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:34.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:34.257048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:34.273 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:34.272422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:44.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:44.385438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:44.419 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:44.418848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:54.484 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:54.483751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:26:54.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:26:54.498549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:04.576 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:04.575968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:04.586 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:04.586383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:14.676 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:14.675657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:14.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:14.686867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:24.787 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:24.786422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:24.801 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:24.801419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:34.871 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:34.869762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:34.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:34.879314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:44.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:44.963480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:44.980 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:44.979764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:55.074 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:55.073314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:27:55.089 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:27:55.089211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:05.170 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:05.168909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:05.186 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:05.185488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:15.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:15.304297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:15.363 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:15.340817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:25.447 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:25.446401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:25.463 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:25.462156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:35.532 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:35.531487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:35.544 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:35.543834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:45.672 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:45.671782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:45.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:45.683881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:55.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:55.769960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:28:55.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:28:55.783159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:05.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:05.891018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:05.905 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:05.904428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:15.964 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:15.964322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:16.016 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:16.015773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:26.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:26.085379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:26.143 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:26.142315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:36.222 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:36.218156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:36.232 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:36.231640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:46.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:46.301944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:46.316 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:46.316037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:56.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:56.387288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:29:56.402 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:29:56.401692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:06.473 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:06.472400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:06.495 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:06.488375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:16.606 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:16.606026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:16.619 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:16.618691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:26.752 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:26.751902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:26.763 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:26.762628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:36.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:36.848312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:36.861 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:36.860801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:46.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:46.953173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:46.968 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:46.968165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:57.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:57.092364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:30:57.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:30:57.104012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:07.190 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:07.189279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:07.202 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:07.202127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:17.294 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:17.293035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:17.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:17.307983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:27.429 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:27.428612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:27.442 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:27.441667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:37.517 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:37.517279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:37.532 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:37.531414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:47.608 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:47.607886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:47.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:47.678148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:57.801 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:57.800760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:31:57.815 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:31:57.814783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:07.973 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:07.972641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:07.985 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:07.985146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:18.069 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:18.068303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:18.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:18.083229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:28.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:28.270468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:28.281 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:28.280929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:38.345 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:38.344247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:38.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:38.359497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:48.435 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:48.431954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:48.457 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:48.443934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:58.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:58.548519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:32:58.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:32:58.560913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:08.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:08.626840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:08.643 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:08.636994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:18.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:18.744127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:18.759 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:18.757112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:28.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:28.878480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:28.887 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:28.887482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:39.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:39.028664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:39.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:39.041637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:49.140 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:49.139216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:49.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:49.153142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:59.262 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:59.261040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:33:59.273 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:33:59.273443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:09.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:09.332176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:09.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:09.346484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:19.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:19.486840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:19.498 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:19.498143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:29.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:29.628890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:29.641 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:29.640661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:39.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:39.709696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:39.722 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:39.721518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:49.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:49.789196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:49.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:49.803329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:59.891 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:59.877391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:34:59.891 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:34:59.888423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:10.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:09.997062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:10.051 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:10.010190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:20.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:20.113870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:20.126 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:20.125847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:30.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:30.234373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:30.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:30.249408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:40.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:40.312498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:40.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:40.324350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:50.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:50.390982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:35:50.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:35:50.404197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:00.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:00.466654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:00.503 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:00.480489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:10.551 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:10.550190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:10.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:10.561114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:20.635 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:20.634129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:20.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:20.685659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:30.814 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:30.813318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:30.827 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:30.826420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:40.922 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:40.921381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:40.935 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:40.934881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:51.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:51.049351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:36:51.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:36:51.061137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:01.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:01.143925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:01.156 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:01.156049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:11.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:11.226492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:11.278 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:11.277103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:21.397 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:21.396452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:21.411 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:21.410792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:31.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:31.477005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:31.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:31.489065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:41.610 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:41.609489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:41.619 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:41.618643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:51.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:51.742539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:37:51.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:37:51.754322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:01.831 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:01.830830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:01.845 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:01.845303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:11.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:11.943108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:11.956 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:11.955510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:22.051 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:22.050897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:22.063 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:22.063220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:32.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:32.126439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:32.139 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:32.138996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:42.252 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:42.249830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:42.260 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:42.259480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:52.337 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:52.336519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:38:52.357 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:38:52.353535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:02.474 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:02.473319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:02.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:02.485071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:12.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:12.561125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:12.574 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:12.574340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:22.640 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:22.636589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:22.706 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:22.705360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:32.793 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:32.793094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:32.806 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:32.805970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:42.925 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:42.924477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:42.938 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:42.937438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:53.059 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:53.058809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:39:53.073 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:39:53.072337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:03.207 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:03.206353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:03.222 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:03.221534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:13.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:13.324004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:13.337 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:13.337298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:14.367 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:14.366590 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T07:40:15.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:15.001004 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T07:40:14.991+0000 7f387940e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T07:40:15.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T07:40:14.991+0000 7f387940e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T07:40:15.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T07:40:14.991+0000 7f387940e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T07:40:15.003 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T07:40:23.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:23.415026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:23.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:23.429884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:33.541 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:33.540834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:33.551 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:33.550621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:43.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:43.668514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:43.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:43.681126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:53.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:53.746119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:40:53.816 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:40:53.815928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:03.894 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:03.893237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:03.907 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:03.907401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:13.971 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:13.971031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:13.985 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:13.984484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:24.087 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:24.086008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:24.121 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:24.120589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:34.190 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:34.189293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:34.203 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:34.202714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:44.298 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:44.297885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:44.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:44.310883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:54.383 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:54.382889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:41:54.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:41:54.393688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:04.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:04.483923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:04.498 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:04.497681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:14.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:14.561426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:14.573 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:14.573145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:24.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:24.683208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:24.693 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:24.692335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:34.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:34.850312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:34.863 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:34.863058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:44.935 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:44.935001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:44.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:44.947560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:55.017 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:55.016332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:42:55.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:42:55.033879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:05.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:05.101606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:05.113 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:05.112717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:15.210 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:15.209483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:15.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:15.222846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:25.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:25.280981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:25.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:25.294722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:35.402 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:35.401742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:35.413 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:35.413232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:45.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:45.534461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:45.548 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:45.547872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:55.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:55.650548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:43:55.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:43:55.666879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:05.768 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:05.768034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:05.787 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:05.780212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:15.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:15.882354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:15.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:15.894140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:25.996 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:25.995907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:26.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:26.006804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:36.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:36.139281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:36.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:36.151302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:46.257 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:46.256903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:46.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:46.272568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:56.345 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:56.344798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:44:56.355 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:44:56.355271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:06.466 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:06.465228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:06.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:06.478710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:16.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:16.587806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:16.601 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:16.600490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:26.662 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:26.660096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:26.709 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:26.708520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:36.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:36.816324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:36.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:36.830345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:46.993 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:46.992249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:47.005 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:47.004711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:57.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:57.094968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:45:57.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:45:57.108233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:07.196 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:07.195135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:07.207 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:07.207162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:17.358 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:17.357988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:17.373 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:17.373050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:27.490 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:27.488914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:27.505 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:27.504319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:37.610 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:37.609559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:37.624 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:37.624280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:47.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:47.745013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:47.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:47.756826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:57.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:57.871511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:46:57.884 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:46:57.883878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:07.996 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:07.995137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:08.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:08.006538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:18.108 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:18.107853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:18.118 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:18.117823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:28.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:28.221974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:28.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:28.235860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:38.308 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:38.307889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:38.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:38.322472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:48.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:48.383258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:48.434 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:48.433938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:58.504 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:58.503604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:47:58.517 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:47:58.516932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:08.615 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:08.614942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:08.632 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:08.626555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:18.732 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:18.731170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:18.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:18.744989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:28.845 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:28.844277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:28.857 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:28.856343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:38.959 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:38.958697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:38.984 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:38.981620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:49.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:49.094234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:49.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:49.106855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:59.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:59.215159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:48:59.229 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:48:59.229098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:09.298 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:09.297386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:09.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:09.310014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:19.427 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:19.426862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:19.440 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:19.439600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:29.551 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:29.550486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:29.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:29.563496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:39.632 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:39.631622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:39.650 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:39.650010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:49.726 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:49.725566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:49.739 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:49.738486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:59.845 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:59.843826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:49:59.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:49:59.857654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:09.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:09.928313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:09.948 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:09.943403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:20.010 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:20.009708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:20.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:20.018858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:30.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:30.115666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:30.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:30.129068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:40.232 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:40.230319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:40.245 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:40.245210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:50.324 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:50.323224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:50:50.337 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:50:50.336922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:00.411 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:00.409818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:00.424 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:00.423210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:10.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:10.476604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:10.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:10.489123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:20.607 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:20.595728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:20.615 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:20.614107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:30.690 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:30.689011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:30.702 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:30.701906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:40.810 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:40.809739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:40.823 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:40.822590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:50.898 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:50.897617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:51:50.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:51:50.909040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:01.023 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:01.022844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:01.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:01.037968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:11.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:11.175134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:11.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:11.187298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:21.267 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:21.266301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:21.280 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:21.279781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:31.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:31.394812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:31.419 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:31.408102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:41.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:41.544515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:41.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:41.559342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:51.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:51.673752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:52:51.696 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:52:51.695573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:01.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:01.794706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:01.807 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:01.806408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:11.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:11.892120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:11.962 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:11.961411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:22.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:22.031556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:22.045 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:22.045087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:32.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:32.152807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:32.167 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:32.166618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:42.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:42.270053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:42.285 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:42.285077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:52.357 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:52.355927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:53:52.371 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:53:52.370989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:02.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:02.500720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:02.514 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:02.513640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:12.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:12.662399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:12.677 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:12.676636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:22.813 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:22.812344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:22.825 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:22.824730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:32.924 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:32.923637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:32.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:32.936188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:43.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:43.018507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:43.035 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:43.031924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:53.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:53.148270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:54:53.162 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:54:53.161411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:03.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:03.236012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:03.249 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:03.249087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:13.326 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:13.325646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:13.339 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:13.339029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:23.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:23.405820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:23.418 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:23.417591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:33.483 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:33.482255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:33.506 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:33.495720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:43.600 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:43.599490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:43.616 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:43.615417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:53.735 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:53.734520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:55:53.749 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:55:53.748861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:03.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:03.865362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:03.878 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:03.877560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:13.983 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:13.983017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:13.999 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:13.999067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:24.126 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:24.126207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:24.138 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:24.138263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:34.212 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:34.212208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:34.224 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:34.223583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:44.290 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:44.288790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:44.299 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:44.298685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:54.398 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:54.397855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:54.413 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:54.411203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:56:55.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:55.011629 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T07:56:55.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:56:55.734503 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T07:56:55.724+0000 7f17694e1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T07:56:55.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T07:56:55.724+0000 7f17694e1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T07:56:55.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T07:56:55.724+0000 7f17694e1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T07:56:55.737 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T07:57:04.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:04.511011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:04.525 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:04.524967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:14.623 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:14.622244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:14.635 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:14.635245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:24.738 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:24.737871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:24.749 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:24.749119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:34.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:34.878909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:34.894 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:34.893662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:44.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:44.964544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:44.977 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:44.976857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:55.047 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:55.045491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:57:55.058 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:57:55.058124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:05.135 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:05.134850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:05.147 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:05.147224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:15.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:15.232342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:15.290 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:15.289943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:25.396 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:25.395355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:25.409 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:25.408493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:35.476 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:35.475823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:35.488 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:35.487730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:45.593 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:45.592828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:45.606 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:45.605739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:55.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:55.668515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:58:55.680 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:58:55.679753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:05.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:05.772889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:05.788 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:05.787766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:15.865 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:15.864736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:15.878 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:15.878461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:25.973 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:25.972548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:25.986 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:25.986197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:36.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:36.093848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:36.108 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:36.107550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:46.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:46.214233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:46.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:46.225517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:56.309 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:56.308385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T07:59:56.321 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 07:59:56.320418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:06.425 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:06.424287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:06.437 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:06.437338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:16.505 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:16.504600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:16.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:16.568808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:26.724 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:26.723889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:26.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:26.736229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:36.915 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:36.914875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:36.927 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:36.926875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:46.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:46.988220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:47.000 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:46.999847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:57.111 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:57.110814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:00:57.125 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:00:57.124505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:07.201 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:07.200301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:07.213 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:07.212949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:17.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:17.301688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:17.318 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:17.317369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:27.493 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:27.492904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:27.507 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:27.506637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:37.582 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:37.580236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:37.593 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:37.593190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:47.712 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:47.711469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:47.752 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:47.751535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:57.821 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:57.820743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:01:57.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:01:57.834483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:07.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:07.945771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:07.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:07.956853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:18.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:18.061162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:18.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:18.076920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:28.152 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:28.151748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:28.213 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:28.210294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:38.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:38.276040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:38.290 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:38.289727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:48.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:48.390793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:48.403 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:48.403003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:58.466 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:58.465236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:02:58.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:02:58.477027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:08.602 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:08.601166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:08.615 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:08.613191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:18.728 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:18.728032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:18.741 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:18.741307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:28.864 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:28.863089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:28.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:28.876134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:38.994 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:38.993860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:39.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:39.007163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:49.108 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:49.107545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:49.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:49.119768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:59.231 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:59.230483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:03:59.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:03:59.243265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:09.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:09.388106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:09.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:09.399950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:19.495 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:19.494165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:19.510 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:19.509227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:29.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:29.629189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:29.643 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:29.643383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:39.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:39.777170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:39.793 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:39.793224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:50.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:50.049652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:04:50.063 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:04:50.063010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:00.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:00.129238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:00.139 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:00.139154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:10.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:10.247307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:10.260 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:10.259846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:20.337 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:20.336140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:20.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:20.387068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:30.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:30.540944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:30.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:30.552142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:40.656 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:40.656151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:40.672 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:40.669479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:50.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:50.751607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:05:50.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:05:50.763604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:00.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:00.829304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:00.842 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:00.841789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:10.913 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:10.912587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:10.942 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:10.942412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:21.014 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:21.013731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:21.046 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:21.045917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:31.152 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:31.151471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:31.163 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:31.162381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:41.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:41.258863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:41.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:41.270875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:51.350 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:51.349652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:06:51.363 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:06:51.363116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:01.464 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:01.463094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:01.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:01.476511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:11.572 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:11.571032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:11.584 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:11.584073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:21.655 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:21.654405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:21.666 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:21.665500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:31.767 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:31.766866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:31.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:31.782909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:41.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:41.873662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:41.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:41.885862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:52.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:52.006082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:07:52.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:07:52.018302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:02.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:02.145452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:02.159 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:02.159368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:12.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:12.235619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:12.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:12.247707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:22.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:22.312833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:22.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:22.324714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:32.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:32.386360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:32.399 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:32.399004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:42.466 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:42.465636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:42.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:42.477846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:52.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:52.568867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:08:52.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:08:52.582652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:02.729 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:02.728822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:02.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:02.743124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:12.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:12.850495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:12.871 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:12.870444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:22.939 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:22.939042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:22.969 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:22.968306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:33.086 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:33.085026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:33.101 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:33.100972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:43.191 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:43.190690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:43.206 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:43.205122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:53.304 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:53.303319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:09:53.317 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:09:53.316442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:03.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:03.385006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:03.398 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:03.397661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:13.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:13.470567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:13.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:13.485102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:23.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:23.588047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:23.626 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:23.626056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:33.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:33.732825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:33.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:33.745972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:43.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:43.832898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:43.846 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:43.846029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:53.932 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:53.931780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:10:53.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:10:53.945583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:04.031 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:04.031051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:04.045 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:04.044989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:14.158 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:14.157477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:14.191 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:14.190226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:24.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:24.301312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:24.317 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:24.315200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:34.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:34.386839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:34.404 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:34.401020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:44.473 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:44.471644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:44.486 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:44.485238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:54.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:54.562636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:11:54.620 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:11:54.619732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:04.728 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:04.727908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:04.741 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:04.741317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:14.846 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:14.845519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:14.856 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:14.855511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:24.917 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:24.913518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:24.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:24.975097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:35.111 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:35.110138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:35.124 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:35.123793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:45.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:45.236499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:45.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:45.248158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:55.362 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:55.361340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:12:55.377 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:12:55.376559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:05.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:05.486540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:05.500 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:05.499454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:15.574 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:15.573642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:15.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:15.588122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:25.688 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:25.687168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:25.701 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:25.701056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:35.741 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:35.740724 I | op-mon: parsing mon endpoints: c=10.110.14.128:6789,a=10.104.177.194:6789,b=10.99.92.113:6789 2022-03-08T08:13:35.789 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:35.788590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:35.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:35.801730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:36.488 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:36.487361 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T08:13:36.473+0000 7f646633f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T08:13:36.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T08:13:36.473+0000 7f646633f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T08:13:36.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08T08:13:36.473+0000 7f646633f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T08:13:36.489 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-08T08:13:45.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:45.873484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:45.887 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:45.886948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:55.958 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:55.957978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:13:55.972 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:13:55.972303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:06.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:06.114993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:06.128 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:06.127364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:16.186 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:16.185366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:16.197 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:16.197095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:26.265 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:26.264304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:26.287 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:26.286615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:36.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:36.384618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:36.399 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:36.398748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:46.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:46.511011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:46.524 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:46.523452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:56.623 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:56.622795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:14:56.639 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:14:56.638390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:06.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:06.715410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:06.730 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:06.729825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:16.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:16.794009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:16.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:16.804152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:26.916 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:26.915067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:26.928 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:26.927721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:37.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:37.011560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:37.025 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:37.024872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:47.126 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:47.125349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:47.142 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:47.137398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:57.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:57.227328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:15:57.242 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:15:57.240481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:07.320 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:07.320087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:07.333 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:07.332568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:17.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:17.455778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:17.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:17.468346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:27.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:27.538866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:27.598 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:27.597946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:37.688 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:37.687789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:37.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:37.739362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:47.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:47.850234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:47.866 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:47.865714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:57.987 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:57.987182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:16:57.997 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:16:57.997057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:08.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:08.105709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:08.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:08.119101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:18.221 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:18.220824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:18.239 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:18.238507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:28.358 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:28.357228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:28.370 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:28.370033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:38.472 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:38.472064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:38.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:38.485470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:48.555 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:48.554276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:48.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:48.567586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:58.648 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:58.647799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:17:58.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:17:58.660229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:08.768 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:08.767149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:08.781 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:08.780852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:18.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:18.885859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:18.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:18.898836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:28.973 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:28.972267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:28.987 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:28.986744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:39.064 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:39.064177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:39.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:39.114570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:49.191 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:49.188976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:49.203 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:49.202977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:59.321 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:59.320223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:18:59.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:18:59.335713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:09.432 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:09.430925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:09.444 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:09.443911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:19.666 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:19.665519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:19.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:19.679046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:29.864 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:29.863440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:29.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:29.875563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:40.080 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:40.079516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:40.091 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:40.091023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:50.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:50.178268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:19:50.194 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:19:50.193318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:00.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:00.309566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:00.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:00.325282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:10.424 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:10.423450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:10.438 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:10.437682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:20.520 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:20.519290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:20.534 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:20.532782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:30.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:30.621934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:30.636 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:30.635604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:40.721 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:40.720479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:40.732 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:40.732005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:50.822 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:50.821939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:20:50.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:20:50.832521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:00.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:00.990334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:01.004 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:01.003682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:11.109 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:11.108775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:11.121 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:11.120434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:21.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:21.255272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:21.270 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:21.269331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:31.382 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:31.381526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:31.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:31.392769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:41.494 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:41.493176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T08:21:41.507 INFO:tasks.rook.operator.smithi018.stdout:2022-03-08 08:21:41.506327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T10:42:36.637 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-08T10:42:36.675 DEBUG:teuthology.task.console_log:Killing console logger for smithi018 2022-03-08T10:42:36.676 DEBUG:teuthology.task.console_log:Killing console logger for smithi084 2022-03-08T10:42:36.677 DEBUG:teuthology.task.console_log:Killing console logger for smithi105 2022-03-08T10:42:36.677 DEBUG:teuthology.exit:Finished running handlers