2022-04-09T11:30:31.082 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-09T11:30:31.084 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi/6783804 --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/1-node k8s/1.21 net/flannel rook/master} --owner scheduled_ozeneva@teuthology --name ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi --verbose -- /home/teuthworker/archive/ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi/6783804/orig.config.yaml 2022-04-09T11:30:31.116 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-04-09T11:30:31.177 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi/6783804 branch: wip-omri-tracing-compiled description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/1-node k8s/1.21 net/flannel rook/master} email: null first_in_suite: false job_id: '6783804' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-omri-tracing-compiled ceph: conf: global: osd crush chooseleaf type: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-omri-tracing-compiled sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 owner: scheduled_ozeneva@teuthology priority: 50 repo: git://git.ceph.com/ceph-ci.git roles: - - host.a - client.a sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 sleep_before_teardown: 0 suite: rados suite_branch: wip-omri-tracing-compiled suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 targets: smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb8AyE8kIp8OME/MFcQijYbpMTU4QIQcU0zV7aKTe/qRSYTBZdbxvLCkOc6twYyDtbpBeWuv/fhIvNMQUZBIK9FkWQO51YMPtEq1LcD0xyg/LNAb5ZP6iOZpoht9BetoBISrjjmzQQwu8tC1cQjbr1/7WD/8c38/nwbgQwzH5E2Pq3sK8hgzy8Q7zZ4bEFWqwWyN+7X9NIhOxTaKgvG46rfH84eEbQr7pI/PWwR1NTL+N9tz/uS8YQyG5o72KKF4d0r5jt5VfD/iUf5NfgC4UQ74waaNnb1X7TmC4S+GKNc7Vj5ll6uUY83c3f0mQO+l8nHZJacloCtkVBEN4IrbEzV1SMiBm3/BO3JHGafoVI9R/UIhHZCPl0hBOxZh6yy5kJbpDMTsgv/csGWVkCIN3szsaIiaEmYEgJ8cR/e3k/pexQ4Js5R4V71gzlFN1GginNQIk9XvfRurrRXc83ZOaPfDtgQmMwEVxe/h4P3yl1AojwZWZvb3O5fwzFAHWWz4k= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: a347b2ac5400a43ba76a678399b1deda7dae1a6c timestamp: 2022-04-09_11:21:29 tube: smithi user: ozeneva verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-04-09T11:30:31.178 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa; will attempt to use it 2022-04-09T11:30:31.178 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks 2022-04-09T11:30:31.179 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-04-09T11:30:31.181 INFO:teuthology.task.internal:Checking packages... 2022-04-09T11:30:31.204 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a70e525ea98f7194c5e516757c6105ca14c81cb7' 2022-04-09T11:30:31.205 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-09T11:30:31.205 INFO:teuthology.packaging:ref: None 2022-04-09T11:30:31.206 INFO:teuthology.packaging:tag: None 2022-04-09T11:30:31.206 INFO:teuthology.packaging:branch: wip-omri-tracing-compiled 2022-04-09T11:30:31.206 INFO:teuthology.packaging:sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 2022-04-09T11:30:31.207 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-omri-tracing-compiled 2022-04-09T11:30:31.370 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11488-ga70e525e-1focal 2022-04-09T11:30:31.371 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-04-09T11:30:31.395 INFO:teuthology.task.internal:no buildpackages task found 2022-04-09T11:30:31.395 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-04-09T11:30:31.467 INFO:teuthology.task.internal:Saving configuration 2022-04-09T11:30:31.483 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-04-09T11:30:31.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-09T11:30:31.625 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi/6783804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-09 11:25:34.623057', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFNH745l+nXtnzhSbDFxpcc9Z4VK0+qZOVqe1uWZM+g8'} 2022-04-09T11:30:31.625 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-04-09T11:30:31.637 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-04-09T11:30:31.637 INFO:teuthology.run_tasks:Running task console_log... 2022-04-09T11:30:31.699 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7d6fb49950>, signals=[15]) 2022-04-09T11:30:31.700 INFO:teuthology.run_tasks:Running task internal.connect... 2022-04-09T11:30:31.721 INFO:teuthology.task.internal:Opening connections... 2022-04-09T11:30:31.721 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2022-04-09T11:30:31.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-09T11:30:31.783 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-04-09T11:30:31.795 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2022-04-09T11:30:31.909 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2022-04-09T11:30:31.910 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2022-04-09T11:30:31.916 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="Ubuntu" 2022-04-09T11:30:31.916 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-04-09T11:30:31.917 INFO:teuthology.orchestra.run.smithi052.stdout:ID=ubuntu 2022-04-09T11:30:31.917 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE=debian 2022-04-09T11:30:31.917 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-04-09T11:30:31.917 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="20.04" 2022-04-09T11:30:31.918 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-09T11:30:31.918 INFO:teuthology.orchestra.run.smithi052.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-09T11:30:31.918 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-09T11:30:31.918 INFO:teuthology.orchestra.run.smithi052.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-09T11:30:31.919 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_CODENAME=focal 2022-04-09T11:30:31.919 INFO:teuthology.orchestra.run.smithi052.stdout:UBUNTU_CODENAME=focal 2022-04-09T11:30:31.920 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2022-04-09T11:30:31.938 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-04-09T11:30:31.952 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-04-09T11:30:32.068 INFO:teuthology.task.internal:Checking for old test directory... 2022-04-09T11:30:32.069 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2022-04-09T11:30:32.075 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-04-09T11:30:32.119 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-04-09T11:30:32.119 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2022-04-09T11:30:32.135 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-04-09T11:30:32.243 INFO:teuthology.run_tasks:Running task kernel... 2022-04-09T11:30:32.300 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-04-09T11:30:32.300 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-04-09T11:30:32.301 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}} 2022-04-09T11:30:32.301 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-04-09T11:30:32.302 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-09T11:30:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-09T11:30:32.308 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2022-04-09T11:30:32.315 INFO:teuthology.orchestra.run.smithi052.stdout:5.4.0-107-generic 2022-04-09T11:30:32.315 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.4.0-107-generic 2022-04-09T11:30:32.316 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get clean 2022-04-09T11:30:32.416 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get update 2022-04-09T11:30:32.561 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-09T11:30:32.602 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-04-09T11:30:32.602 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-04-09T11:30:32.650 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-04-09T11:30:32.932 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [629 kB] 2022-04-09T11:30:33.019 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,712 kB] 2022-04-09T11:30:33.049 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [320 kB] 2022-04-09T11:30:33.053 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.9 kB] 2022-04-09T11:30:33.053 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [917 kB] 2022-04-09T11:30:33.065 INFO:teuthology.orchestra.run.smithi052.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [131 kB] 2022-04-09T11:30:33.066 INFO:teuthology.orchestra.run.smithi052.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [417 kB] 2022-04-09T11:30:33.066 INFO:teuthology.orchestra.run.smithi052.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [676 kB] 2022-04-09T11:30:33.074 INFO:teuthology.orchestra.run.smithi052.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [916 kB] 2022-04-09T11:30:33.084 INFO:teuthology.orchestra.run.smithi052.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [204 kB] 2022-04-09T11:30:33.085 INFO:teuthology.orchestra.run.smithi052.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.4 kB] 2022-04-09T11:30:33.138 INFO:teuthology.orchestra.run.smithi052.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,384 kB] 2022-04-09T11:30:33.181 INFO:teuthology.orchestra.run.smithi052.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [239 kB] 2022-04-09T11:30:33.184 INFO:teuthology.orchestra.run.smithi052.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,884 B] 2022-04-09T11:30:33.184 INFO:teuthology.orchestra.run.smithi052.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [859 kB] 2022-04-09T11:30:33.194 INFO:teuthology.orchestra.run.smithi052.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [122 kB] 2022-04-09T11:30:33.196 INFO:teuthology.orchestra.run.smithi052.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [696 kB] 2022-04-09T11:30:33.204 INFO:teuthology.orchestra.run.smithi052.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [547 kB] 2022-04-09T11:30:33.210 INFO:teuthology.orchestra.run.smithi052.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [122 kB] 2022-04-09T11:30:33.212 INFO:teuthology.orchestra.run.smithi052.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.1 kB] 2022-04-09T11:30:36.804 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 10.3 MB in 2s (6,024 kB/s) 2022-04-09T11:30:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2022-04-09T11:30:38.076 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-09T11:30:38.143 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2022-04-09T11:30:38.324 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2022-04-09T11:30:38.325 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2022-04-09T11:30:38.484 INFO:teuthology.orchestra.run.smithi052.stdout:The following additional packages will be installed: 2022-04-09T11:30:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-09T11:30:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-modules-5.13.0-39-generic 2022-04-09T11:30:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: linux-modules-extra-5.13.0-39-generic thermald upower usbmuxd 2022-04-09T11:30:38.486 INFO:teuthology.orchestra.run.smithi052.stdout:Suggested packages: 2022-04-09T11:30:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-04-09T11:30:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-39-generic 2022-04-09T11:30:38.539 INFO:teuthology.orchestra.run.smithi052.stdout:The following NEW packages will be installed: 2022-04-09T11:30:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-09T11:30:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-image-generic-hwe-20.04 2022-04-09T11:30:38.540 INFO:teuthology.orchestra.run.smithi052.stdout: linux-modules-5.13.0-39-generic linux-modules-extra-5.13.0-39-generic 2022-04-09T11:30:38.541 INFO:teuthology.orchestra.run.smithi052.stdout: thermald upower usbmuxd 2022-04-09T11:30:38.720 INFO:teuthology.orchestra.run.smithi052.stdout:0 upgraded, 13 newly installed, 0 to remove and 8 not upgraded. 2022-04-09T11:30:38.720 INFO:teuthology.orchestra.run.smithi052.stdout:Need to get 79.9 MB of archives. 2022-04-09T11:30:38.721 INFO:teuthology.orchestra.run.smithi052.stdout:After this operation, 410 MB of additional disk space will be used. 2022-04-09T11:30:38.721 INFO:teuthology.orchestra.run.smithi052.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-04-09T11:30:38.986 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-04-09T11:30:39.013 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-04-09T11:30:39.031 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-04-09T11:30:39.103 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-04-09T11:30:39.138 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [18.3 MB] 2022-04-09T11:30:39.932 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [10.0 MB] 2022-04-09T11:30:40.184 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [51.0 MB] 2022-04-09T11:30:41.776 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.39.44~20.04.24 [2,620 B] 2022-04-09T11:30:41.777 INFO:teuthology.orchestra.run.smithi052.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-04-09T11:30:41.777 INFO:teuthology.orchestra.run.smithi052.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-04-09T11:30:41.782 INFO:teuthology.orchestra.run.smithi052.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-04-09T11:30:41.843 INFO:teuthology.orchestra.run.smithi052.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-04-09T11:30:42.269 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 79.9 MB in 3s (24.2 MB/s) 2022-04-09T11:30:42.433 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-04-09T11:30:43.966 INFO:teuthology.orchestra.run.smithi052.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102172 files and directories currently installed.) 2022-04-09T11:30:43.967 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-04-09T11:30:44.096 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-09T11:30:44.449 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libplist3:amd64. 2022-04-09T11:30:44.463 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-04-09T11:30:44.499 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-04-09T11:30:44.844 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-04-09T11:30:44.858 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-04-09T11:30:44.902 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-09T11:30:45.256 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-04-09T11:30:45.268 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-04-09T11:30:45.305 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-09T11:30:45.625 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-04-09T11:30:45.640 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-04-09T11:30:45.675 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-09T11:30:45.986 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package linux-modules-5.13.0-39-generic. 2022-04-09T11:30:46.002 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../05-linux-modules-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-09T11:30:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:30:48.520 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package linux-image-5.13.0-39-generic. 2022-04-09T11:30:48.534 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../06-linux-image-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-09T11:30:48.628 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:30:49.115 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-39-generic. 2022-04-09T11:30:49.127 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-09T11:30:49.164 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:30:55.157 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-04-09T11:30:55.173 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.39.44~20.04.24_amd64.deb ... 2022-04-09T11:30:55.215 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-09T11:30:55.551 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libevdev2:amd64. 2022-04-09T11:30:55.568 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-04-09T11:30:55.609 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-09T11:30:55.929 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package thermald. 2022-04-09T11:30:55.945 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-04-09T11:30:55.979 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-04-09T11:30:56.559 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package upower. 2022-04-09T11:30:56.574 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-04-09T11:30:56.627 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking upower (0.99.11-1build2) ... 2022-04-09T11:30:57.005 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package usbmuxd. 2022-04-09T11:30:57.018 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-04-09T11:30:57.063 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-09T11:30:57.336 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-04-09T11:30:57.484 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-09T11:30:57.610 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-09T11:30:57.736 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-09T11:30:57.862 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-09T11:30:58.030 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-09T11:30:58.156 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up upower (0.99.11-1build2) ... 2022-04-09T11:30:58.571 INFO:teuthology.orchestra.run.smithi052.stdout:upower.service is a disabled or a static unit, not starting it. 2022-04-09T11:30:58.708 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-09T11:30:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-04-09T11:30:58.887 INFO:teuthology.orchestra.run.smithi052.stdout:Adding system user `usbmux' (UID 114) ... 2022-04-09T11:30:58.887 INFO:teuthology.orchestra.run.smithi052.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-04-09T11:30:59.382 INFO:teuthology.orchestra.run.smithi052.stdout:Not creating home directory `/var/lib/usbmux'. 2022-04-09T11:30:59.608 INFO:teuthology.orchestra.run.smithi052.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-04-09T11:30:59.654 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-04-09T11:30:59.980 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-04-09T11:30:59.980 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-04-09T11:31:00.449 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:31:01.853 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:31:03.281 INFO:teuthology.orchestra.run.smithi052.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-39-generic 2022-04-09T11:31:03.281 INFO:teuthology.orchestra.run.smithi052.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-39-generic 2022-04-09T11:31:03.420 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:31:04.813 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-09T11:31:04.949 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.7) ... 2022-04-09T11:31:09.803 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-09T11:31:10.430 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-04-09T11:31:10.531 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:31:10.811 INFO:teuthology.orchestra.run.smithi052.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-04-09T11:31:10.811 INFO:teuthology.orchestra.run.smithi052.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-39-generic 2022-04-09T11:31:35.074 INFO:teuthology.orchestra.run.smithi052.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-04-09T11:31:35.074 INFO:teuthology.orchestra.run.smithi052.stdout:Sourcing file `/etc/default/grub' 2022-04-09T11:31:35.080 INFO:teuthology.orchestra.run.smithi052.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-09T11:31:35.101 INFO:teuthology.orchestra.run.smithi052.stdout:Generating grub configuration file ... 2022-04-09T11:31:35.486 INFO:teuthology.orchestra.run.smithi052.stdout:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-09T11:31:35.522 INFO:teuthology.orchestra.run.smithi052.stdout:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-09T11:31:35.954 INFO:teuthology.orchestra.run.smithi052.stdout:Found linux image: /boot/vmlinuz-5.4.0-107-generic 2022-04-09T11:31:35.966 INFO:teuthology.orchestra.run.smithi052.stdout:Found initrd image: /boot/initrd.img-5.4.0-107-generic 2022-04-09T11:31:38.578 INFO:teuthology.orchestra.run.smithi052.stdout:done 2022-04-09T11:31:38.979 DEBUG:teuthology.orchestra.run.smithi052:> dpkg -s linux-image-generic-hwe-20.04 2022-04-09T11:31:39.011 INFO:teuthology.orchestra.run.smithi052.stdout:Package: linux-image-generic-hwe-20.04 2022-04-09T11:31:39.011 INFO:teuthology.orchestra.run.smithi052.stdout:Status: install ok installed 2022-04-09T11:31:39.011 INFO:teuthology.orchestra.run.smithi052.stdout:Priority: optional 2022-04-09T11:31:39.012 INFO:teuthology.orchestra.run.smithi052.stdout:Section: kernel 2022-04-09T11:31:39.012 INFO:teuthology.orchestra.run.smithi052.stdout:Installed-Size: 19 2022-04-09T11:31:39.012 INFO:teuthology.orchestra.run.smithi052.stdout:Maintainer: Ubuntu Kernel Team 2022-04-09T11:31:39.012 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: amd64 2022-04-09T11:31:39.013 INFO:teuthology.orchestra.run.smithi052.stdout:Source: linux-meta-hwe-5.13 2022-04-09T11:31:39.013 INFO:teuthology.orchestra.run.smithi052.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-09T11:31:39.013 INFO:teuthology.orchestra.run.smithi052.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-09T11:31:39.013 INFO:teuthology.orchestra.run.smithi052.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-09T11:31:39.013 INFO:teuthology.orchestra.run.smithi052.stdout:Recommends: thermald 2022-04-09T11:31:39.014 INFO:teuthology.orchestra.run.smithi052.stdout:Description: Generic Linux kernel image 2022-04-09T11:31:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: This package will always depend on the latest generic kernel image 2022-04-09T11:31:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: available. 2022-04-09T11:31:39.014 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-107-generic Expected: 5.13.0-39-generic 2022-04-09T11:31:39.015 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-04-09T11:31:39.015 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-04-09T11:31:39.015 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-04-09T11:31:39.016 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2022-04-09T11:31:39.016 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get clean 2022-04-09T11:31:39.067 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get update 2022-04-09T11:31:39.209 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-09T11:31:39.218 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-09T11:31:39.238 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-09T11:31:39.339 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-09T11:31:40.811 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2022-04-09T11:31:40.834 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-09T11:31:40.900 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2022-04-09T11:31:41.084 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2022-04-09T11:31:41.084 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2022-04-09T11:31:41.281 INFO:teuthology.orchestra.run.smithi052.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.39.44~20.04.24). 2022-04-09T11:31:41.281 INFO:teuthology.orchestra.run.smithi052.stdout:0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded. 2022-04-09T11:31:41.283 DEBUG:teuthology.orchestra.run.smithi052:> dpkg -s linux-image-generic-hwe-20.04 2022-04-09T11:31:41.301 INFO:teuthology.orchestra.run.smithi052.stdout:Package: linux-image-generic-hwe-20.04 2022-04-09T11:31:41.302 INFO:teuthology.orchestra.run.smithi052.stdout:Status: install ok installed 2022-04-09T11:31:41.302 INFO:teuthology.orchestra.run.smithi052.stdout:Priority: optional 2022-04-09T11:31:41.302 INFO:teuthology.orchestra.run.smithi052.stdout:Section: kernel 2022-04-09T11:31:41.303 INFO:teuthology.orchestra.run.smithi052.stdout:Installed-Size: 19 2022-04-09T11:31:41.303 INFO:teuthology.orchestra.run.smithi052.stdout:Maintainer: Ubuntu Kernel Team 2022-04-09T11:31:41.303 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: amd64 2022-04-09T11:31:41.303 INFO:teuthology.orchestra.run.smithi052.stdout:Source: linux-meta-hwe-5.13 2022-04-09T11:31:41.304 INFO:teuthology.orchestra.run.smithi052.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-09T11:31:41.304 INFO:teuthology.orchestra.run.smithi052.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-09T11:31:41.304 INFO:teuthology.orchestra.run.smithi052.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-09T11:31:41.304 INFO:teuthology.orchestra.run.smithi052.stdout:Recommends: thermald 2022-04-09T11:31:41.305 INFO:teuthology.orchestra.run.smithi052.stdout:Description: Generic Linux kernel image 2022-04-09T11:31:41.305 INFO:teuthology.orchestra.run.smithi052.stdout: This package will always depend on the latest generic kernel image 2022-04-09T11:31:41.305 INFO:teuthology.orchestra.run.smithi052.stdout: available. 2022-04-09T11:31:41.306 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2022-04-09T11:31:41.351 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.CzBZsEbICw 2022-04-09T11:31:41.352 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub/grub.cfg /tmp/tmp.CzBZsEbICw 2022-04-09T11:31:41.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.CzBZsEbICw 2022-04-09T11:31:41.461 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.CzBZsEbICw is 8KB 2022-04-09T11:31:41.475 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.CzBZsEbICw 2022-04-09T11:31:41.482 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-04-09T11:31:41.539 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:31:41.540 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-04-09T11:31:41.540 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-04-09T11:31:41.598 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-39-generic 2022-04-09T11:31:41.599 DEBUG:teuthology.orchestra.run.smithi052:> sudo update-grub 2022-04-09T11:31:42.835 INFO:teuthology.orchestra.run.smithi052.stderr:Sourcing file `/etc/default/grub' 2022-04-09T11:31:42.846 INFO:teuthology.orchestra.run.smithi052.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-09T11:31:42.859 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2022-04-09T11:31:43.163 INFO:teuthology.orchestra.run.smithi052.stderr:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-09T11:31:43.176 INFO:teuthology.orchestra.run.smithi052.stderr:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-09T11:31:43.576 INFO:teuthology.orchestra.run.smithi052.stderr:Found linux image: /boot/vmlinuz-5.4.0-107-generic 2022-04-09T11:31:43.589 INFO:teuthology.orchestra.run.smithi052.stderr:Found initrd image: /boot/initrd.img-5.4.0-107-generic 2022-04-09T11:31:45.201 INFO:teuthology.orchestra.run.smithi052.stderr:done 2022-04-09T11:31:45.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2022-04-09T11:32:15.234 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-04-09T11:32:15.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2022-04-09T11:32:15.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-09T11:33:15.295 DEBUG:teuthology.orchestra.remote:timed out 2022-04-09T11:33:45.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2022-04-09T11:33:45.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-09T11:33:45.556 DEBUG:teuthology.orchestra.run.smithi052:> true 2022-04-09T11:33:46.076 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2022-04-09T11:33:46.076 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-39-generic"... 2022-04-09T11:33:46.077 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2022-04-09T11:33:46.084 INFO:teuthology.orchestra.run.smithi052.stdout:5.13.0-39-generic 2022-04-09T11:33:46.085 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-39-generic vs 5.13.0-39-generic 2022-04-09T11:33:46.085 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-04-09T11:33:46.086 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-04-09T11:33:47.086 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-04-09T11:33:47.087 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-04-09T11:33:47.166 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2022-04-09T11:33:47.188 DEBUG:teuthology.parallel:result is None 2022-04-09T11:33:47.188 INFO:teuthology.run_tasks:Running task internal.base... 2022-04-09T11:33:47.196 INFO:teuthology.task.internal:Creating test directory... 2022-04-09T11:33:47.196 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-09T11:33:47.204 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-04-09T11:33:47.231 INFO:teuthology.run_tasks:Running task internal.archive... 2022-04-09T11:33:47.300 INFO:teuthology.task.internal:Creating archive directory... 2022-04-09T11:33:47.301 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-09T11:33:47.322 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-04-09T11:33:47.375 INFO:teuthology.task.internal:Enabling coredump saving... 2022-04-09T11:33:47.376 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-09T11:33:47.398 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-09T11:33:47.408 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-09T11:33:47.410 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-04-09T11:33:47.490 INFO:teuthology.task.internal:Configuring sudo... 2022-04-09T11:33:47.490 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-09T11:33:47.507 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-04-09T11:33:47.586 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-04-09T11:33:47.587 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-09T11:33:47.596 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-09T11:33:47.601 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-09T11:33:47.652 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:33:47.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-09T11:33:47.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2022-04-09T11:33:47.715 INFO:teuthology.run_tasks:Running task internal.timer... 2022-04-09T11:33:47.726 INFO:teuthology.task.internal:Starting timer... 2022-04-09T11:33:47.726 INFO:teuthology.run_tasks:Running task pcp... 2022-04-09T11:33:47.796 INFO:teuthology.run_tasks:Running task selinux... 2022-04-09T11:33:47.876 DEBUG:teuthology.task.selinux:Excluding smithi052: OS 'ubuntu' does not support SELinux 2022-04-09T11:33:47.877 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-04-09T11:33:47.878 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-04-09T11:33:47.879 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-04-09T11:33:47.879 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-04-09T11:33:47.941 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-04-09T11:33:47.942 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-04-09T11:33:47.969 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-04-09T11:33:47.972 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-04-09T11:36:36.665 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com')] 2022-04-09T11:36:36.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2022-04-09T11:36:36.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-09T11:36:36.717 DEBUG:teuthology.orchestra.run.smithi052:> true 2022-04-09T11:36:36.840 INFO:teuthology.run_tasks:Running task clock... 2022-04-09T11:36:36.852 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-04-09T11:36:36.853 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-09T11:36:36.855 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-09T11:36:36.885 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-04-09T11:36:36.887 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: Command line: ntpd -gq 2022-04-09T11:36:36.887 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: proto: precision = 0.078 usec (-24) 2022-04-09T11:36:36.888 INFO:teuthology.orchestra.run.smithi052.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-09T11:36:36.889 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-09T11:36:36.889 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: restrict ::: KOD does nothing without LIMITED. 2022-04-09T11:36:36.890 INFO:teuthology.orchestra.run.smithi052.stderr:restrict ::: KOD does nothing without LIMITED. 2022-04-09T11:36:36.891 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: Listen and drop on 0 v6wildcard [::]:123 2022-04-09T11:36:36.891 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-04-09T11:36:36.891 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: Listen normally on 2 lo 127.0.0.1:123 2022-04-09T11:36:36.891 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: Listen normally on 3 ens1f0 172.21.15.52:123 2022-04-09T11:36:36.892 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: Listen normally on 4 lo [::1]:123 2022-04-09T11:36:36.892 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15de%4]:123 2022-04-09T11:36:36.893 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:36 ntpd[7783]: Listening on routing socket on fd #22 for interface updates 2022-04-09T11:36:37.890 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:37 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:37.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:37 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:37.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:37 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:37.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:37 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:37.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:37 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:37.892 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:37 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:38.886 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:38 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:38.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:38 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:38.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:38 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:38.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:38 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:38.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:38 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:38.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:38 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:39.886 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:39 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:39.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:39 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:39.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:39 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:39.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:39 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:39.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:39 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:39.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:39 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:40.886 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:40 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:40.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:40 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:40.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:40 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:40.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:40 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:40.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:40 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:40.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:40 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:41.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:41.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:41.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:41.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:41.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:41.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:41.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:41.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:41.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:41.890 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:41.890 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:41.890 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:41.890 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:41.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:41.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:41.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:41.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:41.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:41 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:42.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:42 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:42.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:42 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:42.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:42 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:42.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:42 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:42.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:42 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:42.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:42 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:43.887 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:43.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:43.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:43.888 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:43.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:43.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:43.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:43.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:43.889 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:43.890 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:43.890 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:43.890 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:43.890 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:43.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:43.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:36:43.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:36:43.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:36:43.891 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:36:43.892 INFO:teuthology.orchestra.run.smithi052.stdout: 9 Apr 11:36:43 ntpd[7783]: ntpd: time slew +0.002792 s 2022-04-09T11:36:43.892 INFO:teuthology.orchestra.run.smithi052.stdout:ntpd: time slew +0.002792s 2022-04-09T11:36:43.893 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-04-09T11:36:43.893 INFO:teuthology.orchestra.run.smithi052.stderr: 9 Apr 11:36:43 ntpd[7783]: can't open /var/log/ntpstats/loopstats.20220409: Permission denied 2022-04-09T11:36:43.956 INFO:teuthology.orchestra.run.smithi052.stdout: remote refid st t when poll reach delay offset jitter 2022-04-09T11:36:43.957 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================== 2022-04-09T11:36:43.957 INFO:teuthology.orchestra.run.smithi052.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-09T11:36:43.957 INFO:teuthology.orchestra.run.smithi052.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-09T11:36:43.957 INFO:teuthology.orchestra.run.smithi052.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-09T11:36:43.958 INFO:teuthology.orchestra.run.smithi052.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-09T11:36:43.958 INFO:teuthology.run_tasks:Running task kubeadm... 2022-04-09T11:36:43.975 INFO:tasks.kubeadm:Kubeadm start 2022-04-09T11:36:43.975 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-04-09T11:36:43.976 DEBUG:teuthology.orchestra.run.smithi052:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-04-09T11:36:44.024 INFO:teuthology.orchestra.run.smithi052.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-04-09T11:36:44.032 INFO:teuthology.orchestra.run.smithi052.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-04-09T11:36:44.038 INFO:teuthology.orchestra.run.smithi052.stdout:net.ipv4.ip_forward = 1 2022-04-09T11:36:44.090 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:36:44.091 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-04-09T11:36:44.106 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:36:44.106 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/docker/daemon.json 2022-04-09T11:36:44.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl restart docker || true 2022-04-09T11:36:45.139 INFO:tasks.kubeadm:Identifying pod subnet 2022-04-09T11:36:45.149 INFO:tasks.kubeadm:172.21.15.52 is in 172.21.0.0/20 at pos 3891 2022-04-09T11:36:45.166 INFO:tasks.kubeadm:Pod subnet: 10.249.152.0/21 2022-04-09T11:36:45.166 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-04-09T11:36:45.167 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-04-09T11:36:45.188 INFO:teuthology.orchestra.run.smithi052.stderr: 2022-04-09T11:36:45.189 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-09T11:36:45.189 INFO:teuthology.orchestra.run.smithi052.stderr: 2022-04-09T11:36:45.425 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-09T11:36:45.426 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-09T11:36:45.494 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-09T11:36:45.574 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-09T11:36:46.960 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2022-04-09T11:36:47.142 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2022-04-09T11:36:47.143 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2022-04-09T11:36:47.233 INFO:teuthology.orchestra.run.smithi052.stdout:8 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-09T11:36:47.247 INFO:teuthology.orchestra.run.smithi052.stderr: 2022-04-09T11:36:47.248 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-09T11:36:47.248 INFO:teuthology.orchestra.run.smithi052.stderr: 2022-04-09T11:36:47.302 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2022-04-09T11:36:47.483 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2022-04-09T11:36:47.484 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2022-04-09T11:36:47.674 INFO:teuthology.orchestra.run.smithi052.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-04-09T11:36:47.675 INFO:teuthology.orchestra.run.smithi052.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-04-09T11:36:47.675 INFO:teuthology.orchestra.run.smithi052.stdout:The following NEW packages will be installed: 2022-04-09T11:36:47.675 INFO:teuthology.orchestra.run.smithi052.stdout: apt-transport-https 2022-04-09T11:36:47.759 INFO:teuthology.orchestra.run.smithi052.stdout:0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded. 2022-04-09T11:36:47.759 INFO:teuthology.orchestra.run.smithi052.stdout:Need to get 4,680 B of archives. 2022-04-09T11:36:47.759 INFO:teuthology.orchestra.run.smithi052.stdout:After this operation, 162 kB of additional disk space will be used. 2022-04-09T11:36:47.760 INFO:teuthology.orchestra.run.smithi052.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-04-09T11:36:47.983 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: unable to initialize frontend: Dialog 2022-04-09T11:36:47.984 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-09T11:36:47.984 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: falling back to frontend: Readline 2022-04-09T11:36:48.002 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: unable to initialize frontend: Readline 2022-04-09T11:36:48.002 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-09T11:36:48.002 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: falling back to frontend: Teletype 2022-04-09T11:36:48.008 INFO:teuthology.orchestra.run.smithi052.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-09T11:36:48.060 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 4,680 B in 0s (61.7 kB/s) 2022-04-09T11:36:48.150 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package apt-transport-https. 2022-04-09T11:36:48.196 INFO:teuthology.orchestra.run.smithi052.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109370 files and directories currently installed.) 2022-04-09T11:36:48.199 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-04-09T11:36:48.240 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-04-09T11:36:48.511 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up apt-transport-https (2.0.6) ... 2022-04-09T11:36:49.009 INFO:teuthology.orchestra.run.smithi052.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-04-09T11:36:49.024 INFO:teuthology.orchestra.run.smithi052.stderr: 2022-04-09T11:36:49.024 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-09T11:36:49.025 INFO:teuthology.orchestra.run.smithi052.stderr: 2022-04-09T11:36:49.220 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-09T11:36:49.221 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-09T11:36:49.261 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-09T11:36:49.299 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-09T11:36:49.627 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-04-09T11:36:49.887 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-04-09T11:36:50.028 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 64.1 kB in 1s (75.0 kB/s) 2022-04-09T11:36:50.906 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2022-04-09T11:36:51.086 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2022-04-09T11:36:51.087 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2022-04-09T11:36:51.182 INFO:teuthology.orchestra.run.smithi052.stdout:8 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-09T11:36:51.197 INFO:teuthology.orchestra.run.smithi052.stderr: 2022-04-09T11:36:51.198 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-09T11:36:51.198 INFO:teuthology.orchestra.run.smithi052.stderr: 2022-04-09T11:36:51.254 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2022-04-09T11:36:51.436 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2022-04-09T11:36:51.437 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2022-04-09T11:36:51.597 INFO:teuthology.orchestra.run.smithi052.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-04-09T11:36:51.598 INFO:teuthology.orchestra.run.smithi052.stdout:bridge-utils set to manually installed. 2022-04-09T11:36:51.598 INFO:teuthology.orchestra.run.smithi052.stdout:The following additional packages will be installed: 2022-04-09T11:36:51.599 INFO:teuthology.orchestra.run.smithi052.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-04-09T11:36:51.599 INFO:teuthology.orchestra.run.smithi052.stdout:Suggested packages: 2022-04-09T11:36:51.599 INFO:teuthology.orchestra.run.smithi052.stdout: nftables 2022-04-09T11:36:51.648 INFO:teuthology.orchestra.run.smithi052.stdout:The following NEW packages will be installed: 2022-04-09T11:36:51.650 INFO:teuthology.orchestra.run.smithi052.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-04-09T11:36:51.723 INFO:teuthology.orchestra.run.smithi052.stdout:0 upgraded, 8 newly installed, 0 to remove and 8 not upgraded. 2022-04-09T11:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout:Need to get 77.7 MB of archives. 2022-04-09T11:36:51.794 INFO:teuthology.orchestra.run.smithi052.stdout:After this operation, 335 MB of additional disk space will be used. 2022-04-09T11:36:51.795 INFO:teuthology.orchestra.run.smithi052.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-04-09T11:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-04-09T11:36:51.796 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-04-09T11:36:55.316 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-04-09T11:36:56.105 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-04-09T11:36:56.826 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-04-09T11:36:57.472 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-04-09T11:36:57.662 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-04-09T11:36:58.046 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: unable to initialize frontend: Dialog 2022-04-09T11:36:58.047 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-09T11:36:58.047 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: falling back to frontend: Readline 2022-04-09T11:36:58.055 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: unable to initialize frontend: Readline 2022-04-09T11:36:58.055 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-09T11:36:58.055 INFO:teuthology.orchestra.run.smithi052.stderr:debconf: falling back to frontend: Teletype 2022-04-09T11:36:58.061 INFO:teuthology.orchestra.run.smithi052.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-09T11:36:58.114 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 77.7 MB in 6s (12.6 MB/s) 2022-04-09T11:36:58.206 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package conntrack. 2022-04-09T11:36:58.251 INFO:teuthology.orchestra.run.smithi052.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109374 files and directories currently installed.) 2022-04-09T11:36:58.254 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-04-09T11:36:58.304 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-04-09T11:36:58.607 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package cri-tools. 2022-04-09T11:36:58.621 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-04-09T11:36:58.665 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-04-09T11:36:59.976 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ebtables. 2022-04-09T11:36:59.992 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-04-09T11:37:00.042 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-04-09T11:37:00.329 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package kubernetes-cni. 2022-04-09T11:37:00.345 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-04-09T11:37:00.390 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-04-09T11:37:01.857 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package socat. 2022-04-09T11:37:01.872 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-04-09T11:37:01.915 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking socat (1.7.3.3-2) ... 2022-04-09T11:37:02.310 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package kubelet. 2022-04-09T11:37:02.323 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-04-09T11:37:02.360 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking kubelet (1.23.5-00) ... 2022-04-09T11:37:04.813 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package kubectl. 2022-04-09T11:37:04.828 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-04-09T11:37:04.871 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking kubectl (1.23.5-00) ... 2022-04-09T11:37:06.073 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package kubeadm. 2022-04-09T11:37:06.087 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-04-09T11:37:06.132 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-04-09T11:37:07.270 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-04-09T11:37:07.394 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up kubectl (1.23.5-00) ... 2022-04-09T11:37:07.520 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-04-09T11:37:08.281 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up socat (1.7.3.3-2) ... 2022-04-09T11:37:08.407 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up cri-tools (1.23.0-00) ... 2022-04-09T11:37:08.558 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-04-09T11:37:08.684 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up kubelet (1.23.5-00) ... 2022-04-09T11:37:08.907 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-04-09T11:37:09.427 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up kubeadm (1.23.5-00) ... 2022-04-09T11:37:09.846 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-09T11:37:10.719 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-04-09T11:37:24.531 INFO:teuthology.orchestra.run.smithi052.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-04-09T11:37:27.758 INFO:teuthology.orchestra.run.smithi052.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-04-09T11:37:29.888 INFO:teuthology.orchestra.run.smithi052.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-04-09T11:37:33.380 INFO:teuthology.orchestra.run.smithi052.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-04-09T11:37:34.871 INFO:teuthology.orchestra.run.smithi052.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-04-09T11:37:40.135 INFO:teuthology.orchestra.run.smithi052.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-04-09T11:37:42.095 INFO:teuthology.orchestra.run.smithi052.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-04-09T11:37:42.100 INFO:tasks.kubeadm:Token: abcdef.1lgj589zxpfa4asv 2022-04-09T11:37:42.101 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): '172.21.15.52'} 2022-04-09T11:37:42.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo kubeadm init --node-name smithi052 --token abcdef.1lgj589zxpfa4asv --pod-network-cidr 10.249.152.0/21 2022-04-09T11:37:42.287 INFO:teuthology.orchestra.run.smithi052.stdout:[init] Using Kubernetes version: v1.23.5 2022-04-09T11:37:42.288 INFO:teuthology.orchestra.run.smithi052.stdout:[preflight] Running pre-flight checks 2022-04-09T11:37:42.594 INFO:teuthology.orchestra.run.smithi052.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-04-09T11:37:42.595 INFO:teuthology.orchestra.run.smithi052.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-04-09T11:37:42.595 INFO:teuthology.orchestra.run.smithi052.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-04-09T11:37:42.912 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-04-09T11:37:42.979 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "ca" certificate and key 2022-04-09T11:37:43.243 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "apiserver" certificate and key 2022-04-09T11:37:43.244 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi052] and IPs [10.96.0.1 172.21.15.52] 2022-04-09T11:37:43.735 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-04-09T11:37:44.054 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-04-09T11:37:44.238 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-04-09T11:37:44.329 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "etcd/ca" certificate and key 2022-04-09T11:37:44.633 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "etcd/server" certificate and key 2022-04-09T11:37:44.634 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi052] and IPs [172.21.15.52 127.0.0.1 ::1] 2022-04-09T11:37:44.705 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "etcd/peer" certificate and key 2022-04-09T11:37:44.705 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi052] and IPs [172.21.15.52 127.0.0.1 ::1] 2022-04-09T11:37:44.814 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-04-09T11:37:45.043 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-04-09T11:37:45.402 INFO:teuthology.orchestra.run.smithi052.stdout:[certs] Generating "sa" key and public key 2022-04-09T11:37:45.403 INFO:teuthology.orchestra.run.smithi052.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-04-09T11:37:45.599 INFO:teuthology.orchestra.run.smithi052.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-04-09T11:37:45.960 INFO:teuthology.orchestra.run.smithi052.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-04-09T11:37:46.226 INFO:teuthology.orchestra.run.smithi052.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-04-09T11:37:46.352 INFO:teuthology.orchestra.run.smithi052.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-04-09T11:37:46.367 INFO:teuthology.orchestra.run.smithi052.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-04-09T11:37:46.377 INFO:teuthology.orchestra.run.smithi052.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-04-09T11:37:46.378 INFO:teuthology.orchestra.run.smithi052.stdout:[kubelet-start] Starting the kubelet 2022-04-09T11:37:46.605 INFO:teuthology.orchestra.run.smithi052.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-04-09T11:37:46.605 INFO:teuthology.orchestra.run.smithi052.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-04-09T11:37:46.608 INFO:teuthology.orchestra.run.smithi052.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-04-09T11:37:46.609 INFO:teuthology.orchestra.run.smithi052.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-04-09T11:37:46.611 INFO:teuthology.orchestra.run.smithi052.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-04-09T11:37:46.613 INFO:teuthology.orchestra.run.smithi052.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-04-09T11:38:03.116 INFO:teuthology.orchestra.run.smithi052.stdout:[apiclient] All control plane components are healthy after 16.502878 seconds 2022-04-09T11:38:03.117 INFO:teuthology.orchestra.run.smithi052.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-04-09T11:38:03.136 INFO:teuthology.orchestra.run.smithi052.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-04-09T11:38:03.136 INFO:teuthology.orchestra.run.smithi052.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-04-09T11:38:03.682 INFO:teuthology.orchestra.run.smithi052.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-04-09T11:38:03.682 INFO:teuthology.orchestra.run.smithi052.stdout:[mark-control-plane] Marking the node smithi052 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-04-09T11:38:03.683 INFO:teuthology.orchestra.run.smithi052.stdout:[mark-control-plane] Marking the node smithi052 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-04-09T11:38:04.197 INFO:teuthology.orchestra.run.smithi052.stdout:[bootstrap-token] Using token: abcdef.1lgj589zxpfa4asv 2022-04-09T11:38:04.197 INFO:teuthology.orchestra.run.smithi052.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-04-09T11:38:04.227 INFO:teuthology.orchestra.run.smithi052.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-04-09T11:38:04.244 INFO:teuthology.orchestra.run.smithi052.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-04-09T11:38:04.312 INFO:teuthology.orchestra.run.smithi052.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-04-09T11:38:04.329 INFO:teuthology.orchestra.run.smithi052.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-04-09T11:38:04.350 INFO:teuthology.orchestra.run.smithi052.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-04-09T11:38:04.394 INFO:teuthology.orchestra.run.smithi052.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-04-09T11:38:04.961 INFO:teuthology.orchestra.run.smithi052.stdout:[addons] Applied essential addon: CoreDNS 2022-04-09T11:38:05.124 INFO:teuthology.orchestra.run.smithi052.stdout:[addons] Applied essential addon: kube-proxy 2022-04-09T11:38:05.126 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:38:05.127 INFO:teuthology.orchestra.run.smithi052.stdout:Your Kubernetes control-plane has initialized successfully! 2022-04-09T11:38:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:38:05.127 INFO:teuthology.orchestra.run.smithi052.stdout:To start using your cluster, you need to run the following as a regular user: 2022-04-09T11:38:05.127 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:38:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: mkdir -p $HOME/.kube 2022-04-09T11:38:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-04-09T11:38:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-04-09T11:38:05.128 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:38:05.129 INFO:teuthology.orchestra.run.smithi052.stdout:Alternatively, if you are the root user, you can run: 2022-04-09T11:38:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:38:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-04-09T11:38:05.129 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:38:05.129 INFO:teuthology.orchestra.run.smithi052.stdout:You should now deploy a pod network to the cluster. 2022-04-09T11:38:05.130 INFO:teuthology.orchestra.run.smithi052.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-04-09T11:38:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-04-09T11:38:05.130 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:38:05.130 INFO:teuthology.orchestra.run.smithi052.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-04-09T11:38:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:38:05.131 INFO:teuthology.orchestra.run.smithi052.stdout:kubeadm join 172.21.15.52:6443 --token abcdef.1lgj589zxpfa4asv \ 2022-04-09T11:38:05.131 INFO:teuthology.orchestra.run.smithi052.stdout: --discovery-token-ca-cert-hash sha256:c42e595f608a6884f8c3476cb930b5ad86e7ecd0576868306344b53a907b7158 2022-04-09T11:38:05.132 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:38:05.133 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-04-09T11:38:05.150 INFO:tasks.kubeadm:Setting up kubectl 2022-04-09T11:38:05.150 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-04-09T11:38:05.164 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:38:05.164 DEBUG:teuthology.orchestra.run.smithi052:> dd of=.kube/config 2022-04-09T11:38:05.170 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:38:05.170 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/root/.kube/config 2022-04-09T11:38:05.221 DEBUG:teuthology.orchestra.run.smithi052:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-04-09T11:38:05.236 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-04-09T11:38:05.236 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2022-04-09T11:38:05.282 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2022-04-09T11:38:05.283 INFO:teuthology.orchestra.run.smithi052.stdout:apiVersion: policy/v1beta1 2022-04-09T11:38:05.283 INFO:teuthology.orchestra.run.smithi052.stdout:kind: PodSecurityPolicy 2022-04-09T11:38:05.283 INFO:teuthology.orchestra.run.smithi052.stdout:metadata: 2022-04-09T11:38:05.283 INFO:teuthology.orchestra.run.smithi052.stdout: name: psp.flannel.unprivileged 2022-04-09T11:38:05.284 INFO:teuthology.orchestra.run.smithi052.stdout: annotations: 2022-04-09T11:38:05.284 INFO:teuthology.orchestra.run.smithi052.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-04-09T11:38:05.284 INFO:teuthology.orchestra.run.smithi052.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-04-09T11:38:05.284 INFO:teuthology.orchestra.run.smithi052.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-04-09T11:38:05.284 INFO:teuthology.orchestra.run.smithi052.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-04-09T11:38:05.285 INFO:teuthology.orchestra.run.smithi052.stdout:spec: 2022-04-09T11:38:05.285 INFO:teuthology.orchestra.run.smithi052.stdout: privileged: false 2022-04-09T11:38:05.285 INFO:teuthology.orchestra.run.smithi052.stdout: volumes: 2022-04-09T11:38:05.285 INFO:teuthology.orchestra.run.smithi052.stdout: - configMap 2022-04-09T11:38:05.285 INFO:teuthology.orchestra.run.smithi052.stdout: - secret 2022-04-09T11:38:05.286 INFO:teuthology.orchestra.run.smithi052.stdout: - emptyDir 2022-04-09T11:38:05.286 INFO:teuthology.orchestra.run.smithi052.stdout: - hostPath 2022-04-09T11:38:05.286 INFO:teuthology.orchestra.run.smithi052.stdout: allowedHostPaths: 2022-04-09T11:38:05.286 INFO:teuthology.orchestra.run.smithi052.stdout: - pathPrefix: "/etc/cni/net.d" 2022-04-09T11:38:05.287 INFO:teuthology.orchestra.run.smithi052.stdout: - pathPrefix: "/etc/kube-flannel" 2022-04-09T11:38:05.287 INFO:teuthology.orchestra.run.smithi052.stdout: - pathPrefix: "/run/flannel" 2022-04-09T11:38:05.287 INFO:teuthology.orchestra.run.smithi052.stdout: readOnlyRootFilesystem: false 2022-04-09T11:38:05.287 INFO:teuthology.orchestra.run.smithi052.stdout: # Users and groups 2022-04-09T11:38:05.287 INFO:teuthology.orchestra.run.smithi052.stdout: runAsUser: 2022-04-09T11:38:05.288 INFO:teuthology.orchestra.run.smithi052.stdout: rule: RunAsAny 2022-04-09T11:38:05.288 INFO:teuthology.orchestra.run.smithi052.stdout: supplementalGroups: 2022-04-09T11:38:05.288 INFO:teuthology.orchestra.run.smithi052.stdout: rule: RunAsAny 2022-04-09T11:38:05.288 INFO:teuthology.orchestra.run.smithi052.stdout: fsGroup: 2022-04-09T11:38:05.288 INFO:teuthology.orchestra.run.smithi052.stdout: rule: RunAsAny 2022-04-09T11:38:05.289 INFO:teuthology.orchestra.run.smithi052.stdout: # Privilege Escalation 2022-04-09T11:38:05.289 INFO:teuthology.orchestra.run.smithi052.stdout: allowPrivilegeEscalation: false 2022-04-09T11:38:05.289 INFO:teuthology.orchestra.run.smithi052.stdout: defaultAllowPrivilegeEscalation: false 2022-04-09T11:38:05.289 INFO:teuthology.orchestra.run.smithi052.stdout: # Capabilities 2022-04-09T11:38:05.290 INFO:teuthology.orchestra.run.smithi052.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-04-09T11:38:05.290 INFO:teuthology.orchestra.run.smithi052.stdout: defaultAddCapabilities: [] 2022-04-09T11:38:05.290 INFO:teuthology.orchestra.run.smithi052.stdout: requiredDropCapabilities: [] 2022-04-09T11:38:05.290 INFO:teuthology.orchestra.run.smithi052.stdout: # Host namespaces 2022-04-09T11:38:05.291 INFO:teuthology.orchestra.run.smithi052.stdout: hostPID: false 2022-04-09T11:38:05.291 INFO:teuthology.orchestra.run.smithi052.stdout: hostIPC: false 2022-04-09T11:38:05.291 INFO:teuthology.orchestra.run.smithi052.stdout: hostNetwork: true 2022-04-09T11:38:05.291 INFO:teuthology.orchestra.run.smithi052.stdout: hostPorts: 2022-04-09T11:38:05.291 INFO:teuthology.orchestra.run.smithi052.stdout: - min: 0 2022-04-09T11:38:05.292 INFO:teuthology.orchestra.run.smithi052.stdout: max: 65535 2022-04-09T11:38:05.292 INFO:teuthology.orchestra.run.smithi052.stdout: # SELinux 2022-04-09T11:38:05.292 INFO:teuthology.orchestra.run.smithi052.stdout: seLinux: 2022-04-09T11:38:05.292 INFO:teuthology.orchestra.run.smithi052.stdout: # SELinux is unused in CaaSP 2022-04-09T11:38:05.292 INFO:teuthology.orchestra.run.smithi052.stdout: rule: 'RunAsAny' 2022-04-09T11:38:05.293 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2022-04-09T11:38:05.293 INFO:teuthology.orchestra.run.smithi052.stdout:kind: ClusterRole 2022-04-09T11:38:05.293 INFO:teuthology.orchestra.run.smithi052.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-04-09T11:38:05.293 INFO:teuthology.orchestra.run.smithi052.stdout:metadata: 2022-04-09T11:38:05.294 INFO:teuthology.orchestra.run.smithi052.stdout: name: flannel 2022-04-09T11:38:05.294 INFO:teuthology.orchestra.run.smithi052.stdout:rules: 2022-04-09T11:38:05.294 INFO:teuthology.orchestra.run.smithi052.stdout:- apiGroups: ['extensions'] 2022-04-09T11:38:05.294 INFO:teuthology.orchestra.run.smithi052.stdout: resources: ['podsecuritypolicies'] 2022-04-09T11:38:05.294 INFO:teuthology.orchestra.run.smithi052.stdout: verbs: ['use'] 2022-04-09T11:38:05.295 INFO:teuthology.orchestra.run.smithi052.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-04-09T11:38:05.295 INFO:teuthology.orchestra.run.smithi052.stdout:- apiGroups: 2022-04-09T11:38:05.295 INFO:teuthology.orchestra.run.smithi052.stdout: - "" 2022-04-09T11:38:05.295 INFO:teuthology.orchestra.run.smithi052.stdout: resources: 2022-04-09T11:38:05.295 INFO:teuthology.orchestra.run.smithi052.stdout: - pods 2022-04-09T11:38:05.296 INFO:teuthology.orchestra.run.smithi052.stdout: verbs: 2022-04-09T11:38:05.296 INFO:teuthology.orchestra.run.smithi052.stdout: - get 2022-04-09T11:38:05.296 INFO:teuthology.orchestra.run.smithi052.stdout:- apiGroups: 2022-04-09T11:38:05.296 INFO:teuthology.orchestra.run.smithi052.stdout: - "" 2022-04-09T11:38:05.297 INFO:teuthology.orchestra.run.smithi052.stdout: resources: 2022-04-09T11:38:05.297 INFO:teuthology.orchestra.run.smithi052.stdout: - nodes 2022-04-09T11:38:05.297 INFO:teuthology.orchestra.run.smithi052.stdout: verbs: 2022-04-09T11:38:05.297 INFO:teuthology.orchestra.run.smithi052.stdout: - list 2022-04-09T11:38:05.297 INFO:teuthology.orchestra.run.smithi052.stdout: - watch 2022-04-09T11:38:05.298 INFO:teuthology.orchestra.run.smithi052.stdout:- apiGroups: 2022-04-09T11:38:05.298 INFO:teuthology.orchestra.run.smithi052.stdout: - "" 2022-04-09T11:38:05.298 INFO:teuthology.orchestra.run.smithi052.stdout: resources: 2022-04-09T11:38:05.298 INFO:teuthology.orchestra.run.smithi052.stdout: - nodes/status 2022-04-09T11:38:05.298 INFO:teuthology.orchestra.run.smithi052.stdout: verbs: 2022-04-09T11:38:05.299 INFO:teuthology.orchestra.run.smithi052.stdout: - patch 2022-04-09T11:38:05.299 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2022-04-09T11:38:05.299 INFO:teuthology.orchestra.run.smithi052.stdout:kind: ClusterRoleBinding 2022-04-09T11:38:05.299 INFO:teuthology.orchestra.run.smithi052.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-04-09T11:38:05.300 INFO:teuthology.orchestra.run.smithi052.stdout:metadata: 2022-04-09T11:38:05.300 INFO:teuthology.orchestra.run.smithi052.stdout: name: flannel 2022-04-09T11:38:05.300 INFO:teuthology.orchestra.run.smithi052.stdout:roleRef: 2022-04-09T11:38:05.300 INFO:teuthology.orchestra.run.smithi052.stdout: apiGroup: rbac.authorization.k8s.io 2022-04-09T11:38:05.300 INFO:teuthology.orchestra.run.smithi052.stdout: kind: ClusterRole 2022-04-09T11:38:05.301 INFO:teuthology.orchestra.run.smithi052.stdout: name: flannel 2022-04-09T11:38:05.301 INFO:teuthology.orchestra.run.smithi052.stdout:subjects: 2022-04-09T11:38:05.301 INFO:teuthology.orchestra.run.smithi052.stdout:- kind: ServiceAccount 2022-04-09T11:38:05.301 INFO:teuthology.orchestra.run.smithi052.stdout: name: flannel 2022-04-09T11:38:05.301 INFO:teuthology.orchestra.run.smithi052.stdout: namespace: kube-system 2022-04-09T11:38:05.302 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2022-04-09T11:38:05.302 INFO:teuthology.orchestra.run.smithi052.stdout:apiVersion: v1 2022-04-09T11:38:05.302 INFO:teuthology.orchestra.run.smithi052.stdout:kind: ServiceAccount 2022-04-09T11:38:05.302 INFO:teuthology.orchestra.run.smithi052.stdout:metadata: 2022-04-09T11:38:05.303 INFO:teuthology.orchestra.run.smithi052.stdout: name: flannel 2022-04-09T11:38:05.303 INFO:teuthology.orchestra.run.smithi052.stdout: namespace: kube-system 2022-04-09T11:38:05.303 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2022-04-09T11:38:05.303 INFO:teuthology.orchestra.run.smithi052.stdout:kind: ConfigMap 2022-04-09T11:38:05.304 INFO:teuthology.orchestra.run.smithi052.stdout:apiVersion: v1 2022-04-09T11:38:05.304 INFO:teuthology.orchestra.run.smithi052.stdout:metadata: 2022-04-09T11:38:05.304 INFO:teuthology.orchestra.run.smithi052.stdout: name: kube-flannel-cfg 2022-04-09T11:38:05.304 INFO:teuthology.orchestra.run.smithi052.stdout: namespace: kube-system 2022-04-09T11:38:05.304 INFO:teuthology.orchestra.run.smithi052.stdout: labels: 2022-04-09T11:38:05.305 INFO:teuthology.orchestra.run.smithi052.stdout: tier: node 2022-04-09T11:38:05.305 INFO:teuthology.orchestra.run.smithi052.stdout: app: flannel 2022-04-09T11:38:05.305 INFO:teuthology.orchestra.run.smithi052.stdout:data: 2022-04-09T11:38:05.305 INFO:teuthology.orchestra.run.smithi052.stdout: cni-conf.json: | 2022-04-09T11:38:05.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2022-04-09T11:38:05.306 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "cbr0", 2022-04-09T11:38:05.306 INFO:teuthology.orchestra.run.smithi052.stdout: "cniVersion": "0.3.1", 2022-04-09T11:38:05.306 INFO:teuthology.orchestra.run.smithi052.stdout: "plugins": [ 2022-04-09T11:38:05.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2022-04-09T11:38:05.307 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "flannel", 2022-04-09T11:38:05.307 INFO:teuthology.orchestra.run.smithi052.stdout: "delegate": { 2022-04-09T11:38:05.307 INFO:teuthology.orchestra.run.smithi052.stdout: "hairpinMode": true, 2022-04-09T11:38:05.307 INFO:teuthology.orchestra.run.smithi052.stdout: "isDefaultGateway": true 2022-04-09T11:38:05.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2022-04-09T11:38:05.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2022-04-09T11:38:05.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2022-04-09T11:38:05.308 INFO:teuthology.orchestra.run.smithi052.stdout: "type": "portmap", 2022-04-09T11:38:05.308 INFO:teuthology.orchestra.run.smithi052.stdout: "capabilities": { 2022-04-09T11:38:05.308 INFO:teuthology.orchestra.run.smithi052.stdout: "portMappings": true 2022-04-09T11:38:05.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2022-04-09T11:38:05.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2022-04-09T11:38:05.309 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2022-04-09T11:38:05.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2022-04-09T11:38:05.310 INFO:teuthology.orchestra.run.smithi052.stdout: net-conf.json: | 2022-04-09T11:38:05.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2022-04-09T11:38:05.310 INFO:teuthology.orchestra.run.smithi052.stdout: "Network": "10.244.0.0/16", 2022-04-09T11:38:05.310 INFO:teuthology.orchestra.run.smithi052.stdout: "Backend": { 2022-04-09T11:38:05.310 INFO:teuthology.orchestra.run.smithi052.stdout: "Type": "vxlan" 2022-04-09T11:38:05.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2022-04-09T11:38:05.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2022-04-09T11:38:05.311 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2022-04-09T11:38:05.311 INFO:teuthology.orchestra.run.smithi052.stdout:apiVersion: apps/v1 2022-04-09T11:38:05.312 INFO:teuthology.orchestra.run.smithi052.stdout:kind: DaemonSet 2022-04-09T11:38:05.312 INFO:teuthology.orchestra.run.smithi052.stdout:metadata: 2022-04-09T11:38:05.312 INFO:teuthology.orchestra.run.smithi052.stdout: name: kube-flannel-ds 2022-04-09T11:38:05.312 INFO:teuthology.orchestra.run.smithi052.stdout: namespace: kube-system 2022-04-09T11:38:05.312 INFO:teuthology.orchestra.run.smithi052.stdout: labels: 2022-04-09T11:38:05.313 INFO:teuthology.orchestra.run.smithi052.stdout: tier: node 2022-04-09T11:38:05.313 INFO:teuthology.orchestra.run.smithi052.stdout: app: flannel 2022-04-09T11:38:05.313 INFO:teuthology.orchestra.run.smithi052.stdout:spec: 2022-04-09T11:38:05.313 INFO:teuthology.orchestra.run.smithi052.stdout: selector: 2022-04-09T11:38:05.313 INFO:teuthology.orchestra.run.smithi052.stdout: matchLabels: 2022-04-09T11:38:05.314 INFO:teuthology.orchestra.run.smithi052.stdout: app: flannel 2022-04-09T11:38:05.314 INFO:teuthology.orchestra.run.smithi052.stdout: template: 2022-04-09T11:38:05.314 INFO:teuthology.orchestra.run.smithi052.stdout: metadata: 2022-04-09T11:38:05.314 INFO:teuthology.orchestra.run.smithi052.stdout: labels: 2022-04-09T11:38:05.314 INFO:teuthology.orchestra.run.smithi052.stdout: tier: node 2022-04-09T11:38:05.315 INFO:teuthology.orchestra.run.smithi052.stdout: app: flannel 2022-04-09T11:38:05.315 INFO:teuthology.orchestra.run.smithi052.stdout: spec: 2022-04-09T11:38:05.315 INFO:teuthology.orchestra.run.smithi052.stdout: affinity: 2022-04-09T11:38:05.315 INFO:teuthology.orchestra.run.smithi052.stdout: nodeAffinity: 2022-04-09T11:38:05.316 INFO:teuthology.orchestra.run.smithi052.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-04-09T11:38:05.316 INFO:teuthology.orchestra.run.smithi052.stdout: nodeSelectorTerms: 2022-04-09T11:38:05.316 INFO:teuthology.orchestra.run.smithi052.stdout: - matchExpressions: 2022-04-09T11:38:05.316 INFO:teuthology.orchestra.run.smithi052.stdout: - key: kubernetes.io/os 2022-04-09T11:38:05.316 INFO:teuthology.orchestra.run.smithi052.stdout: operator: In 2022-04-09T11:38:05.317 INFO:teuthology.orchestra.run.smithi052.stdout: values: 2022-04-09T11:38:05.317 INFO:teuthology.orchestra.run.smithi052.stdout: - linux 2022-04-09T11:38:05.317 INFO:teuthology.orchestra.run.smithi052.stdout: hostNetwork: true 2022-04-09T11:38:05.317 INFO:teuthology.orchestra.run.smithi052.stdout: priorityClassName: system-node-critical 2022-04-09T11:38:05.317 INFO:teuthology.orchestra.run.smithi052.stdout: tolerations: 2022-04-09T11:38:05.318 INFO:teuthology.orchestra.run.smithi052.stdout: - operator: Exists 2022-04-09T11:38:05.318 INFO:teuthology.orchestra.run.smithi052.stdout: effect: NoSchedule 2022-04-09T11:38:05.318 INFO:teuthology.orchestra.run.smithi052.stdout: serviceAccountName: flannel 2022-04-09T11:38:05.318 INFO:teuthology.orchestra.run.smithi052.stdout: initContainers: 2022-04-09T11:38:05.319 INFO:teuthology.orchestra.run.smithi052.stdout: - name: install-cni-plugin 2022-04-09T11:38:05.319 INFO:teuthology.orchestra.run.smithi052.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-04-09T11:38:05.319 INFO:teuthology.orchestra.run.smithi052.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-04-09T11:38:05.319 INFO:teuthology.orchestra.run.smithi052.stdout: command: 2022-04-09T11:38:05.319 INFO:teuthology.orchestra.run.smithi052.stdout: - cp 2022-04-09T11:38:05.320 INFO:teuthology.orchestra.run.smithi052.stdout: args: 2022-04-09T11:38:05.320 INFO:teuthology.orchestra.run.smithi052.stdout: - -f 2022-04-09T11:38:05.320 INFO:teuthology.orchestra.run.smithi052.stdout: - /flannel 2022-04-09T11:38:05.320 INFO:teuthology.orchestra.run.smithi052.stdout: - /opt/cni/bin/flannel 2022-04-09T11:38:05.321 INFO:teuthology.orchestra.run.smithi052.stdout: volumeMounts: 2022-04-09T11:38:05.321 INFO:teuthology.orchestra.run.smithi052.stdout: - name: cni-plugin 2022-04-09T11:38:05.321 INFO:teuthology.orchestra.run.smithi052.stdout: mountPath: /opt/cni/bin 2022-04-09T11:38:05.321 INFO:teuthology.orchestra.run.smithi052.stdout: - name: install-cni 2022-04-09T11:38:05.321 INFO:teuthology.orchestra.run.smithi052.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-04-09T11:38:05.322 INFO:teuthology.orchestra.run.smithi052.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-04-09T11:38:05.322 INFO:teuthology.orchestra.run.smithi052.stdout: command: 2022-04-09T11:38:05.322 INFO:teuthology.orchestra.run.smithi052.stdout: - cp 2022-04-09T11:38:05.322 INFO:teuthology.orchestra.run.smithi052.stdout: args: 2022-04-09T11:38:05.322 INFO:teuthology.orchestra.run.smithi052.stdout: - -f 2022-04-09T11:38:05.323 INFO:teuthology.orchestra.run.smithi052.stdout: - /etc/kube-flannel/cni-conf.json 2022-04-09T11:38:05.323 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 120k 0 --:--:-- --:--:-- --:--:-- 120k 2022-04-09T11:38:05.324 INFO:teuthology.orchestra.run.smithi052.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-04-09T11:38:05.325 INFO:teuthology.orchestra.run.smithi052.stdout: volumeMounts: 2022-04-09T11:38:05.325 INFO:teuthology.orchestra.run.smithi052.stdout: - name: cni 2022-04-09T11:38:05.325 INFO:teuthology.orchestra.run.smithi052.stdout: mountPath: /etc/cni/net.d 2022-04-09T11:38:05.326 INFO:teuthology.orchestra.run.smithi052.stdout: - name: flannel-cfg 2022-04-09T11:38:05.326 INFO:teuthology.orchestra.run.smithi052.stdout: mountPath: /etc/kube-flannel/ 2022-04-09T11:38:05.326 INFO:teuthology.orchestra.run.smithi052.stdout: containers: 2022-04-09T11:38:05.326 INFO:teuthology.orchestra.run.smithi052.stdout: - name: kube-flannel 2022-04-09T11:38:05.326 INFO:teuthology.orchestra.run.smithi052.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-04-09T11:38:05.327 INFO:teuthology.orchestra.run.smithi052.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-04-09T11:38:05.327 INFO:teuthology.orchestra.run.smithi052.stdout: command: 2022-04-09T11:38:05.327 INFO:teuthology.orchestra.run.smithi052.stdout: - /opt/bin/flanneld 2022-04-09T11:38:05.327 INFO:teuthology.orchestra.run.smithi052.stdout: args: 2022-04-09T11:38:05.328 INFO:teuthology.orchestra.run.smithi052.stdout: - --ip-masq 2022-04-09T11:38:05.328 INFO:teuthology.orchestra.run.smithi052.stdout: - --kube-subnet-mgr 2022-04-09T11:38:05.328 INFO:teuthology.orchestra.run.smithi052.stdout: resources: 2022-04-09T11:38:05.328 INFO:teuthology.orchestra.run.smithi052.stdout: requests: 2022-04-09T11:38:05.329 INFO:teuthology.orchestra.run.smithi052.stdout: cpu: "100m" 2022-04-09T11:38:05.329 INFO:teuthology.orchestra.run.smithi052.stdout: memory: "50Mi" 2022-04-09T11:38:05.330 INFO:teuthology.orchestra.run.smithi052.stdout: limits: 2022-04-09T11:38:05.330 INFO:teuthology.orchestra.run.smithi052.stdout: cpu: "100m" 2022-04-09T11:38:05.330 INFO:teuthology.orchestra.run.smithi052.stdout: memory: "50Mi" 2022-04-09T11:38:05.330 INFO:teuthology.orchestra.run.smithi052.stdout: securityContext: 2022-04-09T11:38:05.331 INFO:teuthology.orchestra.run.smithi052.stdout: privileged: false 2022-04-09T11:38:05.331 INFO:teuthology.orchestra.run.smithi052.stdout: capabilities: 2022-04-09T11:38:05.331 INFO:teuthology.orchestra.run.smithi052.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-04-09T11:38:05.331 INFO:teuthology.orchestra.run.smithi052.stdout: env: 2022-04-09T11:38:05.331 INFO:teuthology.orchestra.run.smithi052.stdout: - name: POD_NAME 2022-04-09T11:38:05.332 INFO:teuthology.orchestra.run.smithi052.stdout: valueFrom: 2022-04-09T11:38:05.332 INFO:teuthology.orchestra.run.smithi052.stdout: fieldRef: 2022-04-09T11:38:05.332 INFO:teuthology.orchestra.run.smithi052.stdout: fieldPath: metadata.name 2022-04-09T11:38:05.332 INFO:teuthology.orchestra.run.smithi052.stdout: - name: POD_NAMESPACE 2022-04-09T11:38:05.333 INFO:teuthology.orchestra.run.smithi052.stdout: valueFrom: 2022-04-09T11:38:05.333 INFO:teuthology.orchestra.run.smithi052.stdout: fieldRef: 2022-04-09T11:38:05.333 INFO:teuthology.orchestra.run.smithi052.stdout: fieldPath: metadata.namespace 2022-04-09T11:38:05.333 INFO:teuthology.orchestra.run.smithi052.stdout: volumeMounts: 2022-04-09T11:38:05.334 INFO:teuthology.orchestra.run.smithi052.stdout: - name: run 2022-04-09T11:38:05.334 INFO:teuthology.orchestra.run.smithi052.stdout: mountPath: /run/flannel 2022-04-09T11:38:05.334 INFO:teuthology.orchestra.run.smithi052.stdout: - name: flannel-cfg 2022-04-09T11:38:05.334 INFO:teuthology.orchestra.run.smithi052.stdout: mountPath: /etc/kube-flannel/ 2022-04-09T11:38:05.334 INFO:teuthology.orchestra.run.smithi052.stdout: - name: xtables-lock 2022-04-09T11:38:05.335 INFO:teuthology.orchestra.run.smithi052.stdout: mountPath: /run/xtables.lock 2022-04-09T11:38:05.335 INFO:teuthology.orchestra.run.smithi052.stdout: volumes: 2022-04-09T11:38:05.335 INFO:teuthology.orchestra.run.smithi052.stdout: - name: run 2022-04-09T11:38:05.335 INFO:teuthology.orchestra.run.smithi052.stdout: hostPath: 2022-04-09T11:38:05.335 INFO:teuthology.orchestra.run.smithi052.stdout: path: /run/flannel 2022-04-09T11:38:05.336 INFO:teuthology.orchestra.run.smithi052.stdout: - name: cni-plugin 2022-04-09T11:38:05.336 INFO:teuthology.orchestra.run.smithi052.stdout: hostPath: 2022-04-09T11:38:05.336 INFO:teuthology.orchestra.run.smithi052.stdout: path: /opt/cni/bin 2022-04-09T11:38:05.336 INFO:teuthology.orchestra.run.smithi052.stdout: - name: cni 2022-04-09T11:38:05.337 INFO:teuthology.orchestra.run.smithi052.stdout: hostPath: 2022-04-09T11:38:05.337 INFO:teuthology.orchestra.run.smithi052.stdout: path: /etc/cni/net.d 2022-04-09T11:38:05.337 INFO:teuthology.orchestra.run.smithi052.stdout: - name: flannel-cfg 2022-04-09T11:38:05.337 INFO:teuthology.orchestra.run.smithi052.stdout: configMap: 2022-04-09T11:38:05.337 INFO:teuthology.orchestra.run.smithi052.stdout: name: kube-flannel-cfg 2022-04-09T11:38:05.338 INFO:teuthology.orchestra.run.smithi052.stdout: - name: xtables-lock 2022-04-09T11:38:05.338 INFO:teuthology.orchestra.run.smithi052.stdout: hostPath: 2022-04-09T11:38:05.338 INFO:teuthology.orchestra.run.smithi052.stdout: path: /run/xtables.lock 2022-04-09T11:38:05.338 INFO:teuthology.orchestra.run.smithi052.stdout: type: FileOrCreate 2022-04-09T11:38:05.366 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-04-09T11:38:05.367 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.249.152.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-04-09T11:38:05.381 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.249.152.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-04-09T11:38:05.381 DEBUG:teuthology.orchestra.run.smithi052:> kubectl apply -f - 2022-04-09T11:38:08.587 INFO:teuthology.orchestra.run.smithi052.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-04-09T11:38:08.591 INFO:teuthology.orchestra.run.smithi052.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-04-09T11:38:08.621 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-04-09T11:38:08.630 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-04-09T11:38:08.659 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/flannel created 2022-04-09T11:38:08.674 INFO:teuthology.orchestra.run.smithi052.stdout:configmap/kube-flannel-cfg created 2022-04-09T11:38:08.714 INFO:teuthology.orchestra.run.smithi052.stdout:daemonset.apps/kube-flannel-ds created 2022-04-09T11:38:08.717 INFO:tasks.kubeadm:Scanning for scratch devices 2022-04-09T11:38:08.717 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:38:08.718 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2022-04-09T11:38:08.723 INFO:tasks.kubeadm:Scratch devices on smithi052: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-09T11:38:08.724 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-04-09T11:38:08.748 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records in 2022-04-09T11:38:08.748 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2022-04-09T11:38:08.749 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111919 s, 937 MB/s 2022-04-09T11:38:08.750 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-04-09T11:38:08.773 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records in 2022-04-09T11:38:08.774 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2022-04-09T11:38:08.774 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130259 s, 805 MB/s 2022-04-09T11:38:08.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-04-09T11:38:08.852 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records in 2022-04-09T11:38:08.852 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2022-04-09T11:38:08.853 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0187155 s, 560 MB/s 2022-04-09T11:38:08.854 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-04-09T11:38:08.879 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records in 2022-04-09T11:38:08.880 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2022-04-09T11:38:08.880 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137123 s, 765 MB/s 2022-04-09T11:38:08.887 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-04-09T11:38:08.887 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi052-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi052 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi052-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi052 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi052-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi052 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi052-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi052 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-04-09T11:38:08.887 DEBUG:teuthology.orchestra.run.smithi052:> kubectl create -f - 2022-04-09T11:38:09.114 INFO:teuthology.orchestra.run.smithi052.stdout:persistentvolume/smithi052-lv-1 created 2022-04-09T11:38:09.140 INFO:teuthology.orchestra.run.smithi052.stdout:persistentvolume/smithi052-lv-2 created 2022-04-09T11:38:09.144 INFO:teuthology.orchestra.run.smithi052.stdout:persistentvolume/smithi052-lv-3 created 2022-04-09T11:38:09.153 INFO:teuthology.orchestra.run.smithi052.stdout:persistentvolume/smithi052-lv-4 created 2022-04-09T11:38:09.161 INFO:teuthology.orchestra.run.smithi052.stdout:storageclass.storage.k8s.io/scratch created 2022-04-09T11:38:09.169 DEBUG:teuthology.orchestra.run.smithi052:> kubectl taint node smithi052 node-role.kubernetes.io/master- || true 2022-04-09T11:38:09.242 INFO:teuthology.orchestra.run.smithi052.stdout:node/smithi052 untainted 2022-04-09T11:38:09.244 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-04-09T11:38:09.244 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-04-09T11:38:09.256 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-04-09T11:38:09.257 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:38:09.257 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2022-04-09T11:38:09.265 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-09T11:38:09.266 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-09T11:38:09.272 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-09T11:38:09.272 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-09T11:38:09.272 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-04-09T11:38:09.273 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-09T11:38:09.273 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2022-04-09 11:38:08.762539598 +0000 2022-04-09T11:38:09.273 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2022-04-09 11:38:08.758539598 +0000 2022-04-09T11:38:09.273 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2022-04-09 11:38:08.758539598 +0000 2022-04-09T11:38:09.274 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2022-04-09T11:38:09.280 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2022-04-09T11:38:09.280 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2022-04-09T11:38:09.281 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000275843 s, 1.9 MB/s 2022-04-09T11:38:09.285 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-09T11:38:09.334 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-09T11:38:09.334 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-09T11:38:09.335 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-04-09T11:38:09.335 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-09T11:38:09.335 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2022-04-09 11:38:08.778539597 +0000 2022-04-09T11:38:09.335 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2022-04-09 11:38:08.778539597 +0000 2022-04-09T11:38:09.336 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2022-04-09 11:38:08.778539597 +0000 2022-04-09T11:38:09.336 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2022-04-09T11:38:09.342 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2022-04-09T11:38:09.343 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2022-04-09T11:38:09.343 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000282878 s, 1.8 MB/s 2022-04-09T11:38:09.347 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-09T11:38:09.354 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-09T11:38:09.355 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-09T11:38:09.355 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-04-09T11:38:09.355 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-09T11:38:09.355 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2022-04-09 11:38:08.854539597 +0000 2022-04-09T11:38:09.356 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2022-04-09 11:38:08.850539597 +0000 2022-04-09T11:38:09.356 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2022-04-09 11:38:08.850539597 +0000 2022-04-09T11:38:09.356 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2022-04-09T11:38:09.363 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2022-04-09T11:38:09.363 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2022-04-09T11:38:09.363 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000324394 s, 1.6 MB/s 2022-04-09T11:38:09.367 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-09T11:38:09.372 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-09T11:38:09.373 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-09T11:38:09.373 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-04-09T11:38:09.373 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-09T11:38:09.373 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2022-04-09 11:38:08.882539596 +0000 2022-04-09T11:38:09.374 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2022-04-09 11:38:08.882539596 +0000 2022-04-09T11:38:09.374 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2022-04-09 11:38:08.882539596 +0000 2022-04-09T11:38:09.374 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2022-04-09T11:38:09.378 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2022-04-09T11:38:09.379 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2022-04-09T11:38:09.379 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.00023353 s, 2.2 MB/s 2022-04-09T11:38:09.383 DEBUG:teuthology.orchestra.run.smithi052:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-04-09T11:38:09.472 INFO:teuthology.orchestra.run.smithi052.stdout:loop 2022-04-09T11:38:09.473 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_1... 2022-04-09T11:38:09.473 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-04-09T11:38:09.494 INFO:teuthology.orchestra.run.smithi052.stdout:1 2022-04-09T11:38:09.520 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_11 2022-04-09T11:38:09.544 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_2... 2022-04-09T11:38:09.545 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-04-09T11:38:09.567 INFO:teuthology.orchestra.run.smithi052.stdout:1 2022-04-09T11:38:09.591 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_21 2022-04-09T11:38:09.612 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_3... 2022-04-09T11:38:09.613 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-04-09T11:38:09.633 INFO:teuthology.orchestra.run.smithi052.stdout:1 2022-04-09T11:38:09.660 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_31 2022-04-09T11:38:09.683 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_4... 2022-04-09T11:38:09.683 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-04-09T11:38:09.702 INFO:teuthology.orchestra.run.smithi052.stdout:1 2022-04-09T11:38:09.726 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_41 2022-04-09T11:38:09.747 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:38:09.748 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2022-04-09T11:38:09.755 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme list 2022-04-09T11:38:09.769 INFO:teuthology.orchestra.run.smithi052.stdout:Node SN Model Namespace Usage Format FW Rev 2022-04-09T11:38:09.769 INFO:teuthology.orchestra.run.smithi052.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-04-09T11:38:09.769 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme0n1 PHMB7513010N480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-04-09T11:38:09.770 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme1n1 8e34a2f7bd0e14bc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:38:09.770 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme2n1 f3ee3cb1a289d60e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:38:09.770 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme3n1 56a38faa62320f83 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:38:09.771 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-04-09T11:38:10.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme list 2022-04-09T11:38:10.787 INFO:teuthology.orchestra.run.smithi052.stdout:Node SN Model Namespace Usage Format FW Rev 2022-04-09T11:38:10.788 INFO:teuthology.orchestra.run.smithi052.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-04-09T11:38:10.788 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme0n1 PHMB7513010N480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-04-09T11:38:10.788 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme1n1 8e34a2f7bd0e14bc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:38:10.788 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme2n1 f3ee3cb1a289d60e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:38:10.789 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme3n1 56a38faa62320f83 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:38:10.789 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme4n1 1f665dabcdcb2355 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:38:10.790 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-04-09T11:38:10.791 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:38:10.791 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/scratch_devs 2022-04-09T11:38:10.804 INFO:teuthology.run_tasks:Running task rook... 2022-04-09T11:38:10.877 INFO:tasks.rook:Rook start 2022-04-09T11:38:10.877 INFO:tasks.rook:Config: {'conf': {'global': {'osd crush chooseleaf type': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a70e525ea98f7194c5e516757c6105ca14c81cb7', 'rook_image': 'rook/ceph:master'} 2022-04-09T11:38:10.878 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7 2022-04-09T11:38:10.878 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-04-09T11:38:10.879 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-04-09T11:38:10.888 INFO:teuthology.orchestra.run.smithi052.stderr:Cloning into 'rook'... 2022-04-09T11:38:13.859 INFO:tasks.rook:/home/teuthworker 2022-04-09T11:38:13.860 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/sbin:/usr/sbin ip addr show 2022-04-09T11:38:13.866 INFO:teuthology.orchestra.run.smithi052.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-04-09T11:38:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-04-09T11:38:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: inet 127.0.0.1/8 scope host lo 2022-04-09T11:38:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2022-04-09T11:38:13.867 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 ::1/128 scope host 2022-04-09T11:38:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2022-04-09T11:38:13.868 INFO:teuthology.orchestra.run.smithi052.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-04-09T11:38:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d2 brd ff:ff:ff:ff:ff:ff 2022-04-09T11:38:13.868 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f0 2022-04-09T11:38:13.869 INFO:teuthology.orchestra.run.smithi052.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-04-09T11:38:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:6b:fd:d3 brd ff:ff:ff:ff:ff:ff 2022-04-09T11:38:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp7s0f1 2022-04-09T11:38:13.869 INFO:teuthology.orchestra.run.smithi052.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-04-09T11:38:13.869 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:de brd ff:ff:ff:ff:ff:ff 2022-04-09T11:38:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f0 2022-04-09T11:38:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: inet 172.21.15.52/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-04-09T11:38:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft 42900sec preferred_lft 42900sec 2022-04-09T11:38:13.870 INFO:teuthology.orchestra.run.smithi052.stdout: inet6 fe80::ec4:7aff:febd:15de/64 scope link 2022-04-09T11:38:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2022-04-09T11:38:13.871 INFO:teuthology.orchestra.run.smithi052.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-04-09T11:38:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 0c:c4:7a:bd:15:df brd ff:ff:ff:ff:ff:ff 2022-04-09T11:38:13.871 INFO:teuthology.orchestra.run.smithi052.stdout: altname enp1s0f1 2022-04-09T11:38:13.872 INFO:teuthology.orchestra.run.smithi052.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-04-09T11:38:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: link/ether 02:42:15:a7:d4:bf brd ff:ff:ff:ff:ff:ff 2022-04-09T11:38:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-04-09T11:38:13.872 INFO:teuthology.orchestra.run.smithi052.stdout: valid_lft forever preferred_lft forever 2022-04-09T11:38:13.873 DEBUG:teuthology.orchestra.run.smithi052:> which systemctl 2022-04-09T11:38:13.880 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/systemctl 2022-04-09T11:38:13.880 DEBUG:teuthology.orchestra.run.smithi052:> true 2022-04-09T11:38:13.955 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2022-04-09T11:38:14.004 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2022-04-09T11:38:14.010 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-09T11:38:14.011 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-04-09T11:38:14.021 DEBUG:teuthology.orchestra.remote:smithi052:rook/deploy/examples/operator.yaml is 20KB 2022-04-09T11:38:14.036 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:38:14.037 DEBUG:teuthology.orchestra.run.smithi052:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-04-09T11:38:14.047 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-04-09T11:38:14.102 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:38:14.103 DEBUG:teuthology.orchestra.run.smithi052:> dd of=operator.yaml 2022-04-09T11:38:14.110 INFO:tasks.rook:Deploying operator 2022-04-09T11:38:14.120 DEBUG:teuthology.orchestra.remote:smithi052:rook/deploy/examples/operator.yaml is 20KB 2022-04-09T11:38:14.146 DEBUG:teuthology.orchestra.remote:smithi052:rook/deploy/examples/operator.yaml is 20KB 2022-04-09T11:38:14.161 DEBUG:teuthology.orchestra.run.smithi052:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-04-09T11:38:14.444 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created 2022-04-09T11:38:14.469 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-04-09T11:38:14.491 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-04-09T11:38:14.568 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-04-09T11:38:14.599 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-04-09T11:38:14.752 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-04-09T11:38:14.775 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-04-09T11:38:14.873 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-04-09T11:38:14.878 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-04-09T11:38:14.908 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-04-09T11:38:14.924 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-04-09T11:38:15.049 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-04-09T11:38:15.076 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-04-09T11:38:15.101 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-04-09T11:38:15.129 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-04-09T11:38:15.172 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-04-09T11:38:15.192 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-04-09T11:38:15.218 INFO:teuthology.orchestra.run.smithi052.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-04-09T11:38:15.267 INFO:teuthology.orchestra.run.smithi052.stdout:namespace/rook-ceph created 2022-04-09T11:38:15.359 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-04-09T11:38:15.377 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-04-09T11:38:15.400 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-04-09T11:38:15.410 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-04-09T11:38:15.418 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-04-09T11:38:15.442 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-04-09T11:38:15.452 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-04-09T11:38:15.526 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-04-09T11:38:15.529 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-04-09T11:38:15.535 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-04-09T11:38:15.543 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-09T11:38:15.552 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-09T11:38:15.560 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-04-09T11:38:15.584 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-04-09T11:38:15.593 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-04-09T11:38:15.602 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-04-09T11:38:15.610 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-04-09T11:38:15.618 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-04-09T11:38:15.668 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-04-09T11:38:15.671 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-09T11:38:15.676 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-09T11:38:15.685 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-04-09T11:38:15.693 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-04-09T11:38:15.701 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-04-09T11:38:15.710 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-04-09T11:38:15.735 INFO:teuthology.orchestra.run.smithi052.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-04-09T11:38:15.743 INFO:teuthology.orchestra.run.smithi052.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-04-09T11:38:15.744 INFO:teuthology.orchestra.run.smithi052.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-04-09T11:38:15.751 INFO:teuthology.orchestra.run.smithi052.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-04-09T11:38:15.760 INFO:teuthology.orchestra.run.smithi052.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-04-09T11:38:15.793 INFO:teuthology.orchestra.run.smithi052.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-04-09T11:38:15.797 INFO:teuthology.orchestra.run.smithi052.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-04-09T11:38:15.818 INFO:teuthology.orchestra.run.smithi052.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-04-09T11:38:15.826 INFO:teuthology.orchestra.run.smithi052.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-09T11:38:15.835 INFO:teuthology.orchestra.run.smithi052.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-04-09T11:38:15.843 INFO:teuthology.orchestra.run.smithi052.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-09T11:38:15.852 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-04-09T11:38:15.860 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-04-09T11:38:15.884 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-04-09T11:38:15.917 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-04-09T11:38:15.920 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-04-09T11:38:15.926 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-04-09T11:38:15.935 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-04-09T11:38:15.943 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-04-09T11:38:15.968 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-04-09T11:38:15.976 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-04-09T11:38:15.985 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-09T11:38:15.993 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-04-09T11:38:16.002 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-04-09T11:38:16.010 INFO:teuthology.orchestra.run.smithi052.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-09T11:38:16.059 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-04-09T11:38:16.063 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-ceph-mgr created 2022-04-09T11:38:16.068 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-ceph-osd created 2022-04-09T11:38:16.076 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-ceph-purge-osd created 2022-04-09T11:38:16.085 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-ceph-rgw created 2022-04-09T11:38:16.119 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-ceph-system created 2022-04-09T11:38:16.145 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-04-09T11:38:16.211 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-04-09T11:38:16.227 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-04-09T11:38:16.252 INFO:teuthology.orchestra.run.smithi052.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-04-09T11:38:16.277 INFO:teuthology.orchestra.run.smithi052.stdout:configmap/rook-ceph-operator-config created 2022-04-09T11:38:16.303 INFO:teuthology.orchestra.run.smithi052.stdout:deployment.apps/rook-ceph-operator created 2022-04-09T11:38:16.309 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:38:16.386 INFO:teuthology.orchestra.run.smithi052.stderr:No resources found in rook-ceph namespace. 2022-04-09T11:38:26.390 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:38:26.454 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:38:26.454 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-operator-75dd789779-rgglw 0/1 Pending 0 8s 2022-04-09T11:38:36.457 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:38:36.516 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:38:36.517 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-operator-75dd789779-rgglw 0/1 ContainerCreating 0 18s 2022-04-09T11:38:46.519 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:38:46.585 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:38:46.585 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-operator-75dd789779-rgglw 0/1 ContainerCreating 0 28s 2022-04-09T11:38:56.588 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:38:56.652 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:38:56.652 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-operator-75dd789779-rgglw 0/1 ContainerCreating 0 38s 2022-04-09T11:39:06.655 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:39:06.838 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:06.839 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-operator-75dd789779-rgglw 0/1 ContainerCreating 0 48s 2022-04-09T11:39:16.842 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:39:16.907 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:16.908 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-operator-75dd789779-rgglw 1/1 Running 0 58s 2022-04-09T11:39:16.910 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-rgglw 2022-04-09T11:39:16.917 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:39:16.918 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2022-04-09T11:39:16.972 INFO:tasks.rook: override: [global] osd crush chooseleaf type = 0 2022-04-09T11:39:16.973 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-04-09T11:39:16.973 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-04-09T11:39:16.973 INFO:tasks.rook: override: [mon] debug mon = 20 2022-04-09T11:39:16.974 INFO:tasks.rook: override: [mon] debug ms = 1 2022-04-09T11:39:16.974 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-04-09T11:39:16.974 INFO:tasks.rook: override: [osd] debug ms = 1 2022-04-09T11:39:16.975 INFO:tasks.rook: override: [osd] debug osd = 20 2022-04-09T11:39:16.975 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-04-09T11:39:16.978 DEBUG:teuthology.orchestra.run.smithi052:> kubectl create -f - 2022-04-09T11:39:17.016 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.099723 I | rookcmd: starting Rook v1.9.0-alpha.0.36.gd23b117de with arguments '/usr/local/bin/rook ceph operator' 2022-04-09T11:39:17.017 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.099833 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-04-09T11:39:17.017 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.099836 I | cephcmd: starting Rook-Ceph operator 2022-04-09T11:39:17.017 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.256735 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-04-09T11:39:17.017 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.263165 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-09T11:39:17.018 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.263186 I | operator: watching all namespaces for Ceph CRs 2022-04-09T11:39:17.018 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.263255 I | operator: setting up schemes 2022-04-09T11:39:17.018 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.266518 I | operator: setting up the controller-runtime manager 2022-04-09T11:39:17.018 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.474369 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-09T11:39:17.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477215 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-09T11:39:17.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477267 I | ceph-cluster-controller: successfully started 2022-04-09T11:39:17.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477344 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-09T11:39:17.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477366 I | ceph-crashcollector-controller: successfully started 2022-04-09T11:39:17.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477393 I | ceph-block-pool-controller: successfully started 2022-04-09T11:39:17.020 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477427 I | ceph-object-store-user-controller: successfully started 2022-04-09T11:39:17.020 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477452 I | ceph-object-realm-controller: successfully started 2022-04-09T11:39:17.020 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477466 I | ceph-object-zonegroup-controller: successfully started 2022-04-09T11:39:17.020 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477484 I | ceph-object-zone-controller: successfully started 2022-04-09T11:39:17.021 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477624 I | ceph-object-controller: successfully started 2022-04-09T11:39:17.021 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477673 I | ceph-file-controller: successfully started 2022-04-09T11:39:17.021 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477709 I | ceph-nfs-controller: successfully started 2022-04-09T11:39:17.021 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477743 I | ceph-rbd-mirror-controller: successfully started 2022-04-09T11:39:17.022 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477775 I | ceph-client-controller: successfully started 2022-04-09T11:39:17.022 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477798 I | ceph-filesystem-mirror-controller: successfully started 2022-04-09T11:39:17.023 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477825 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-09T11:39:17.023 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477850 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-09T11:39:17.023 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477881 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-09T11:39:17.023 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477910 I | ceph-bucket-topic: successfully started 2022-04-09T11:39:17.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477926 I | ceph-bucket-notification: successfully started 2022-04-09T11:39:17.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477942 I | ceph-bucket-notification: successfully started 2022-04-09T11:39:17.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477962 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-09T11:39:17.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.477988 I | blockpool-rados-namespace-controller: successfully started 2022-04-09T11:39:17.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.479016 I | operator: starting the controller-runtime manager 2022-04-09T11:39:17.025 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.582134 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-09T11:39:17.025 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.582175 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-09T11:39:17.025 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.582196 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-09T11:39:17.025 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.586706 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-09T11:39:17.026 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.593613 I | ceph-csi: CSI Ceph RBD driver disabled 2022-04-09T11:39:17.026 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.593642 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-04-09T11:39:17.026 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.595643 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-04-09T11:39:17.026 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.601867 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-04-09T11:39:17.026 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.601886 I | ceph-csi: CSI CephFS driver disabled 2022-04-09T11:39:17.027 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.601904 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-04-09T11:39:17.027 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.603966 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-04-09T11:39:17.027 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:10.609651 I | ceph-csi: successfully removed CSI CephFS driver 2022-04-09T11:39:17.835 INFO:teuthology.orchestra.run.smithi052.stdout:configmap/rook-config-override created 2022-04-09T11:39:17.839 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 1 skipUpgradeChecks: true 2022-04-09T11:39:17.840 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:39:17.840 DEBUG:teuthology.orchestra.run.smithi052:> dd of=cluster.yaml 2022-04-09T11:39:17.846 DEBUG:teuthology.orchestra.run.smithi052:> kubectl create -f cluster.yaml 2022-04-09T11:39:20.000 INFO:teuthology.orchestra.run.smithi052.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-04-09T11:39:20.007 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.010596 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-04-09T11:39:20.016 DEBUG:teuthology.orchestra.remote:smithi052:rook/deploy/examples/operator.yaml is 20KB 2022-04-09T11:39:20.027 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.031682 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:39:20.028 DEBUG:teuthology.orchestra.run.smithi052:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-04-09T11:39:20.031 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.034475 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-04-09T11:39:20.043 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.047344 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-04-09T11:39:20.048 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.052371 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:39:20.082 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084804 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-04-09T11:39:20.082 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084830 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-04-09T11:39:20.082 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084837 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-04-09T11:39:20.083 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084842 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-04-09T11:39:20.083 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084850 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-04-09T11:39:20.083 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084858 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.6.0" (default) 2022-04-09T11:39:20.083 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084865 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-04-09T11:39:20.083 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084872 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-04-09T11:39:20.084 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084880 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-04-09T11:39:20.084 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084887 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-04-09T11:39:20.084 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084894 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-04-09T11:39:20.084 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084903 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-04-09T11:39:20.085 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084921 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-04-09T11:39:20.085 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084928 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-04-09T11:39:20.085 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084934 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-04-09T11:39:20.085 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.084942 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.6.0" 2022-04-09T11:39:20.085 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.085028 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:39:20.086 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:39:20.085046 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:39:20.347 INFO:teuthology.orchestra.run.smithi052.stdout:deployment.apps/rook-ceph-tools created 2022-04-09T11:39:20.352 INFO:tasks.rook:Waiting for tools container to start 2022-04-09T11:39:20.353 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:39:20.517 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:20.517 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 0s 2022-04-09T11:39:25.521 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:39:25.578 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:25.579 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 5s 2022-04-09T11:39:30.582 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:39:31.175 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:31.175 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 11s 2022-04-09T11:39:36.178 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:39:36.246 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:36.246 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 16s 2022-04-09T11:39:41.249 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:39:41.317 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:41.318 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 21s 2022-04-09T11:39:46.320 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:39:46.389 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:46.389 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 26s 2022-04-09T11:39:51.393 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:39:51.462 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:51.462 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 31s 2022-04-09T11:39:56.465 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:39:56.532 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:39:56.532 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 36s 2022-04-09T11:40:01.534 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:40:01.602 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:40:01.602 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 41s 2022-04-09T11:40:03.949 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949222 I | ceph-csi: Detected ceph CSI image version: "v3.6.0" 2022-04-09T11:40:03.950 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949258 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-04-09T11:40:03.950 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949268 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-09T11:40:03.950 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949273 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-09T11:40:03.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949278 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-09T11:40:03.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949283 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-09T11:40:03.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949288 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-09T11:40:03.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949293 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-09T11:40:03.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949298 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-09T11:40:03.952 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949441 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-09T11:40:03.952 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949451 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-09T11:40:03.952 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949462 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-09T11:40:03.952 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949467 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:40:03.952 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949472 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:40:03.953 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949477 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-04-09T11:40:03.953 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949482 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-04-09T11:40:03.953 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949487 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-04-09T11:40:03.953 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949492 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-04-09T11:40:03.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949508 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-04-09T11:40:03.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949515 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:40:03.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949521 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:40:03.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949526 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-04-09T11:40:03.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949530 I | ceph-csi: Kubernetes version is 1.23 2022-04-09T11:40:03.955 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949537 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-04-09T11:40:03.955 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.949542 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-04-09T11:40:03.970 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.971433 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:40:03.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.971456 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:40:03.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.971461 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:40:03.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.971464 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:40:03.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.971467 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:40:03.972 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.971476 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:40:03.972 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:03.971479 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-04-09T11:40:04.168 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:04.169546 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:40:04.168 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:04.169567 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:40:04.169 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:04.169570 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:40:05.014 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:05.014510 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-04-09T11:40:06.345 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:06.344138 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:40:06.345 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:06.344167 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:40:06.346 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:06.344174 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-04-09T11:40:06.605 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:40:07.151 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:07.049548 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:40:07.151 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:07.049575 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:40:07.151 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:07.049582 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:40:07.763 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:40:07.764 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 47s 2022-04-09T11:40:08.342 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:08.340935 I | ceph-csi: successfully started CSI CephFS driver 2022-04-09T11:40:09.499 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:09.475706 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-09T11:40:09.785 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:09.784615 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-04-09T11:40:09.786 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:09.784743 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-09T11:40:10.281 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:10.280461 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-04-09T11:40:12.768 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:40:13.092 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:40:13.092 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 53s 2022-04-09T11:40:16.329 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:16.327700 I | ceph-spec: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:40:16.329 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:16.327729 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:40:16.371 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:16.369968 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:40:16.574 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:16.571538 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-04-09 11:40:16.505044484 +0000 UTC m=+66.426628459 LastTransitionTime:2022-04-09 11:40:16.505044368 +0000 UTC m=+66.426628355}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:40:16.715 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:16.714451 I | op-mon: start running mons 2022-04-09T11:40:16.799 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:16.798960 I | op-mon: creating mon secrets for a new cluster 2022-04-09T11:40:17.206 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:16.860890 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-04-09T11:40:17.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:16.864916 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-04-09T11:40:17.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:17.020268 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:40:17.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:17.020488 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:40:18.096 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:40:18.217 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:40:18.218 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 58s 2022-04-09T11:40:18.581 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:18.580410 I | op-mon: targeting the mon count 1 2022-04-09T11:40:18.593 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:18.593144 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-04-09T11:40:18.986 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:18.984997 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi052 2022-04-09T11:40:18.986 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:18.985021 I | op-mon: mon a assigned to node smithi052 2022-04-09T11:40:18.990 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:18.989007 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-04-09T11:40:18.993 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:18.993188 I | op-mon: creating mon a 2022-04-09T11:40:19.273 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:19.272706 I | op-mon: mon "a" ip is 10.97.79.62 2022-04-09T11:40:19.839 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:19.785119 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:-1] 2022-04-09T11:40:19.840 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:19.785204 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-04-09T11:40:19.840 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:19.785273 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-04-09T11:40:20.587 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:20.586589 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:40:20.616 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:20.587727 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:40:21.120 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:21.119935 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:40:21.123 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:21.119973 I | op-mon: updating obsolete maxMonID -1 to actual value 0 2022-04-09T11:40:21.123 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:21.120024 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-04-09T11:40:21.123 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:21.120045 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:40:21.127 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:21.121944 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-04-09T11:40:21.127 INFO:tasks.rook.operator.smithi052.stdout:I0409 11:40:21.122025 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:40:21.376 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:21.374851 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-04-09T11:40:21.449 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:21.448497 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-04-09T11:40:22.306 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:22.305443 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:40:22.306 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:22.305469 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:40:22.492 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:22.491256 I | op-mon: mon a is not yet running 2022-04-09T11:40:22.492 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:22.491281 I | op-mon: mons running: [] 2022-04-09T11:40:23.220 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:40:23.297 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:40:23.297 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 63s 2022-04-09T11:40:28.300 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:40:28.437 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:40:28.437 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 0/1 ContainerCreating 0 68s 2022-04-09T11:40:33.440 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:40:33.967 INFO:teuthology.orchestra.run.smithi052.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:40:33.967 INFO:teuthology.orchestra.run.smithi052.stdout:rook-ceph-tools-d6d7c985c-mtmfz 1/1 Running 0 73s 2022-04-09T11:40:33.969 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-04-09T11:40:33.970 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch status -f json 2022-04-09T11:40:42.638 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:42.637932 I | op-mon: mons running: [a] 2022-04-09T11:40:53.079 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:53.078170 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:40:53.079 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:53.078204 I | op-mon: mons created: 1 2022-04-09T11:40:53.299 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:53.298275 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:40:53.306 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:53.306116 I | op-mon: mons running: [a] 2022-04-09T11:40:53.523 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:53.522879 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:40:53.523 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:53.522916 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:40:53.638 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:40:53.639 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2022-04-09T11:40:53.639 INFO:teuthology.orchestra.run.smithi052.stderr:pg stat 2022-04-09T11:40:53.639 INFO:teuthology.orchestra.run.smithi052.stderr:pg getmap 2022-04-09T11:40:53.640 INFO:teuthology.orchestra.run.smithi052.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-09T11:40:53.640 INFO:teuthology.orchestra.run.smithi052.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-09T11:40:53.640 INFO:teuthology.orchestra.run.smithi052.stderr:pg dump_pools_json 2022-04-09T11:40:53.641 INFO:teuthology.orchestra.run.smithi052.stderr:pg ls-by-pool [...] 2022-04-09T11:40:53.641 INFO:teuthology.orchestra.run.smithi052.stderr:pg ls-by-primary [] [...] 2022-04-09T11:40:53.641 INFO:teuthology.orchestra.run.smithi052.stderr:pg ls-by-osd [] [...] 2022-04-09T11:40:53.641 INFO:teuthology.orchestra.run.smithi052.stderr:pg ls [] [...] 2022-04-09T11:40:53.642 INFO:teuthology.orchestra.run.smithi052.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-09T11:40:53.642 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2022-04-09T11:40:53.642 INFO:teuthology.orchestra.run.smithi052.stderr:command terminated with exit code 22 2022-04-09T11:40:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-09T11:40:53.871 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:53.870689 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:40:53.871 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:53.870738 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:40:54.161 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:54.161270 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:40:54.162 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:54.161309 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:40:54.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:54.448420 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:40:54.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:54.448456 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:40:54.754 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:54.753376 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:40:54.754 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:54.753413 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:40:54.975 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:54.974242 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:40:54.975 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:54.974288 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:40:55.270 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:55.269530 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:40:55.639 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:55.638963 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:40:55.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:55.920214 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:40:56.262 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:56.261750 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:40:56.262 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:56.261781 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:40:56.546 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:56.545622 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:40:56.547 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:56.545646 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:40:56.547 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:56.545651 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:40:56.839 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:56.839038 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:40:56.864 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:56.863585 I | op-mgr: start running mgr 2022-04-09T11:40:56.864 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:40:56.863613 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:41:03.647 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch status -f json 2022-04-09T11:41:04.216 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:41:04.216 INFO:teuthology.orchestra.run.smithi052.stderr:no valid command found; 10 closest matches: 2022-04-09T11:41:04.216 INFO:teuthology.orchestra.run.smithi052.stderr:pg stat 2022-04-09T11:41:04.217 INFO:teuthology.orchestra.run.smithi052.stderr:pg getmap 2022-04-09T11:41:04.217 INFO:teuthology.orchestra.run.smithi052.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-09T11:41:04.217 INFO:teuthology.orchestra.run.smithi052.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-09T11:41:04.217 INFO:teuthology.orchestra.run.smithi052.stderr:pg dump_pools_json 2022-04-09T11:41:04.218 INFO:teuthology.orchestra.run.smithi052.stderr:pg ls-by-pool [...] 2022-04-09T11:41:04.218 INFO:teuthology.orchestra.run.smithi052.stderr:pg ls-by-primary [] [...] 2022-04-09T11:41:04.218 INFO:teuthology.orchestra.run.smithi052.stderr:pg ls-by-osd [] [...] 2022-04-09T11:41:04.218 INFO:teuthology.orchestra.run.smithi052.stderr:pg ls [] [...] 2022-04-09T11:41:04.218 INFO:teuthology.orchestra.run.smithi052.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-09T11:41:04.219 INFO:teuthology.orchestra.run.smithi052.stderr:Error EINVAL: invalid command 2022-04-09T11:41:04.220 INFO:teuthology.orchestra.run.smithi052.stderr:command terminated with exit code 22 2022-04-09T11:41:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-09T11:41:14.224 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch status -f json 2022-04-09T11:41:14.784 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:41:14.785 INFO:teuthology.orchestra.run.smithi052.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-04-09T11:41:14.790 INFO:teuthology.orchestra.run.smithi052.stderr:command terminated with exit code 2 2022-04-09T11:41:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-04-09T11:41:18.440 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.440103 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-04-09T11:41:18.445 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.445256 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:41:18.520 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.519355 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-04-09T11:41:18.520 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.519451 I | op-mgr: successful modules: balancer 2022-04-09T11:41:18.548 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.548296 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:41:18.549 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.548326 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-04-09T11:41:18.549 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.548338 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:41:18.605 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.605111 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:41:18.612 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.612102 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-04-09T11:41:18.612 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.612128 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:41:18.612 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:18.612134 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:41:19.052 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:19.051157 I | op-mgr: successful modules: prometheus 2022-04-09T11:41:19.056 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:19.056218 I | op-mgr: successful modules: dashboard 2022-04-09T11:41:19.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:19.195394 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-09T11:41:19.196 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:19.195422 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:41:19.209 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:19.209268 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:41:19.209 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:19.209299 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:41:19.210 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:19.209308 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:41:20.156 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:20.156427 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:41:24.794 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch status -f json 2022-04-09T11:41:35.056 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:35.056126 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:41:35.078 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:41:35.079 INFO:teuthology.orchestra.run.smithi052.stdout:{"available": true, "backend": "rook"} 2022-04-09T11:41:35.100 INFO:tasks.rook: mgr/rook orchestrator is active 2022-04-09T11:41:35.101 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph config set mgr mgr/rook/storage_class scratch 2022-04-09T11:41:35.689 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch apply osd --all-available-devices 2022-04-09T11:41:36.344 INFO:teuthology.orchestra.run.smithi052.stdout:['Success'] 2022-04-09T11:41:36.346 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.346250 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-09T11:41:36.346 INFO:tasks.rook.operator.smithi052.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", AllowUnsupported: true}, 2022-04-09T11:41:36.346 INFO:tasks.rook.operator.smithi052.stdout: Storage: v1.StorageScopeSpec{ 2022-04-09T11:41:36.347 INFO:tasks.rook.operator.smithi052.stdout: ... // 3 identical fields 2022-04-09T11:41:36.351 INFO:tasks.rook.operator.smithi052.stdout: Config: nil, 2022-04-09T11:41:36.351 INFO:tasks.rook.operator.smithi052.stdout: Selection: {}, 2022-04-09T11:41:36.351 INFO:tasks.rook.operator.smithi052.stdout:- StorageClassDeviceSets: nil, 2022-04-09T11:41:36.352 INFO:tasks.rook.operator.smithi052.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-04-09T11:41:36.352 INFO:tasks.rook.operator.smithi052.stdout:+ { 2022-04-09T11:41:36.353 INFO:tasks.rook.operator.smithi052.stdout:+ Name: "smithi052-lv-1", 2022-04-09T11:41:36.353 INFO:tasks.rook.operator.smithi052.stdout:+ Count: 1, 2022-04-09T11:41:36.354 INFO:tasks.rook.operator.smithi052.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-09T11:41:36.354 INFO:tasks.rook.operator.smithi052.stdout:+ }, 2022-04-09T11:41:36.355 INFO:tasks.rook.operator.smithi052.stdout:+ { 2022-04-09T11:41:36.355 INFO:tasks.rook.operator.smithi052.stdout:+ Name: "smithi052-lv-2", 2022-04-09T11:41:36.356 INFO:tasks.rook.operator.smithi052.stdout:+ Count: 1, 2022-04-09T11:41:36.356 INFO:tasks.rook.operator.smithi052.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-09T11:41:36.357 INFO:tasks.rook.operator.smithi052.stdout:+ }, 2022-04-09T11:41:36.357 INFO:tasks.rook.operator.smithi052.stdout:+ { 2022-04-09T11:41:36.357 INFO:tasks.rook.operator.smithi052.stdout:+ Name: "smithi052-lv-3", 2022-04-09T11:41:36.358 INFO:tasks.rook.operator.smithi052.stdout:+ Count: 1, 2022-04-09T11:41:36.358 INFO:tasks.rook.operator.smithi052.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-09T11:41:36.358 INFO:tasks.rook.operator.smithi052.stdout:+ }, 2022-04-09T11:41:36.359 INFO:tasks.rook.operator.smithi052.stdout:+ { 2022-04-09T11:41:36.359 INFO:tasks.rook.operator.smithi052.stdout:+ Name: "smithi052-lv-4", 2022-04-09T11:41:36.360 INFO:tasks.rook.operator.smithi052.stdout:+ Count: 1, 2022-04-09T11:41:36.360 INFO:tasks.rook.operator.smithi052.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-09T11:41:36.361 INFO:tasks.rook.operator.smithi052.stdout:+ }, 2022-04-09T11:41:36.361 INFO:tasks.rook.operator.smithi052.stdout:+ }, 2022-04-09T11:41:36.361 INFO:tasks.rook.operator.smithi052.stdout: }, 2022-04-09T11:41:36.361 INFO:tasks.rook.operator.smithi052.stdout: Annotations: nil, 2022-04-09T11:41:36.362 INFO:tasks.rook.operator.smithi052.stdout: Labels: nil, 2022-04-09T11:41:36.362 INFO:tasks.rook.operator.smithi052.stdout: ... // 20 identical fields 2022-04-09T11:41:36.362 INFO:tasks.rook.operator.smithi052.stdout: } 2022-04-09T11:41:36.362 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.346377 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-04-09T11:41:36.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.346803 I | operator: successfully started the controller-runtime manager 2022-04-09T11:41:36.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.346878 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-04-09T11:41:36.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.346993 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-04-09T11:41:36.363 INFO:tasks.rook.operator.smithi052.stdout:I0409 11:41:36.347025 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-04-09T11:41:36.364 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.347289 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-04-09T11:41:36.365 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.364972 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-09T11:41:36.366 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.365003 I | operator: watching all namespaces for Ceph CRs 2022-04-09T11:41:36.366 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.365049 I | operator: setting up schemes 2022-04-09T11:41:36.367 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.367689 I | operator: setting up the controller-runtime manager 2022-04-09T11:41:36.372 INFO:tasks.rook:Waiting for 4 OSDs 2022-04-09T11:41:36.372 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:41:36.573 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.572624 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-09T11:41:36.579 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.575847 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-09T11:41:36.579 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.575893 I | ceph-cluster-controller: successfully started 2022-04-09T11:41:36.579 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.575946 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-09T11:41:36.581 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.575974 I | ceph-crashcollector-controller: successfully started 2022-04-09T11:41:36.581 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576023 I | ceph-block-pool-controller: successfully started 2022-04-09T11:41:36.581 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576053 I | ceph-object-store-user-controller: successfully started 2022-04-09T11:41:36.582 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576081 I | ceph-object-realm-controller: successfully started 2022-04-09T11:41:36.582 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576098 I | ceph-object-zonegroup-controller: successfully started 2022-04-09T11:41:36.582 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576107 I | ceph-object-zone-controller: successfully started 2022-04-09T11:41:36.583 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576222 I | ceph-object-controller: successfully started 2022-04-09T11:41:36.583 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576249 I | ceph-file-controller: successfully started 2022-04-09T11:41:36.583 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576284 I | ceph-nfs-controller: successfully started 2022-04-09T11:41:36.583 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576310 I | ceph-rbd-mirror-controller: successfully started 2022-04-09T11:41:36.584 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576351 I | ceph-client-controller: successfully started 2022-04-09T11:41:36.584 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576367 I | ceph-filesystem-mirror-controller: successfully started 2022-04-09T11:41:36.584 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576384 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-09T11:41:36.584 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576394 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-09T11:41:36.584 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576408 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-09T11:41:36.585 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576427 I | ceph-bucket-topic: successfully started 2022-04-09T11:41:36.585 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576438 I | ceph-bucket-notification: successfully started 2022-04-09T11:41:36.585 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576445 I | ceph-bucket-notification: successfully started 2022-04-09T11:41:36.585 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576453 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-09T11:41:36.586 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.576460 I | blockpool-rados-namespace-controller: successfully started 2022-04-09T11:41:36.586 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.577702 I | operator: starting the controller-runtime manager 2022-04-09T11:41:36.682 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.681395 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-09T11:41:36.682 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.681454 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-09T11:41:36.682 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.681471 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-09T11:41:36.689 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.687523 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-09T11:41:36.698 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.689654 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:41:36.698 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.692409 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:41:36.699 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.692474 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-04-09T11:41:36.699 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.692490 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:41:36.700 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.694710 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:41:36.701 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.696889 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-04-09T11:41:36.701 INFO:tasks.rook.operator.smithi052.stdout:I0409 11:41:36.696979 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:41:36.715 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.715833 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-04-09T11:41:36.719 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.718118 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:41:36.720 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.718162 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:41:36.721 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.718177 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:41:36.724 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721474 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-04-09T11:41:36.724 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721557 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-04-09T11:41:36.725 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721576 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-04-09T11:41:36.725 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721587 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-04-09T11:41:36.725 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721597 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-04-09T11:41:36.725 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721615 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.6.0" (default) 2022-04-09T11:41:36.726 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721625 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-04-09T11:41:36.726 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721633 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-04-09T11:41:36.726 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721641 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-04-09T11:41:36.726 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721650 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-04-09T11:41:36.727 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721656 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-04-09T11:41:36.727 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721664 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-04-09T11:41:36.727 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721674 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-04-09T11:41:36.727 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721681 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-04-09T11:41:36.727 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721688 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-04-09T11:41:36.728 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721695 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.6.0" 2022-04-09T11:41:36.728 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721783 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:41:36.728 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.721798 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:41:36.731 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:36.729458 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:41:36.942 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:41:36.960 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-09T11:41:36.961 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:41:45.098 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094496 I | ceph-csi: Detected ceph CSI image version: "v3.6.0" 2022-04-09T11:41:45.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094542 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-04-09T11:41:45.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094566 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-09T11:41:45.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094591 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-09T11:41:45.100 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094599 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-09T11:41:45.100 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094605 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-09T11:41:45.101 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094623 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-09T11:41:45.101 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094642 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-09T11:41:45.102 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094657 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-09T11:41:45.102 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094669 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-09T11:41:45.102 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094679 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-09T11:41:45.103 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094688 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-09T11:41:45.103 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094696 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:41:45.104 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094706 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:41:45.104 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094715 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-04-09T11:41:45.105 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094729 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-04-09T11:41:45.105 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094757 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-04-09T11:41:45.105 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094768 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-04-09T11:41:45.106 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094777 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-04-09T11:41:45.106 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094788 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:41:45.107 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094798 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:41:45.107 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094807 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-04-09T11:41:45.108 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094816 I | ceph-csi: Kubernetes version is 1.23 2022-04-09T11:41:45.108 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094827 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-04-09T11:41:45.108 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.094836 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-04-09T11:41:45.110 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.105808 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:41:45.111 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.105829 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:41:45.111 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.105833 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:41:45.112 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.105835 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:41:45.112 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.105838 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:41:45.113 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.105840 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:41:45.113 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.105843 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-04-09T11:41:45.795 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.795254 I | ceph-spec: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:41:45.796 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.795285 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:41:45.877 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.877316 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:41:45.877 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.877348 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:41:45.878 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.877365 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:41:45.998 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:45.998616 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:41:46.002 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.002405 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:41:46.003 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.003043 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:41:46.179 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.179307 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-04-09T11:41:46.450 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.450507 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:41:46.450 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.450532 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:41:46.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.450538 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-04-09T11:41:46.460 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.461298 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:41:46.953 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.953442 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:41:46.953 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.953477 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:41:46.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:46.953486 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:41:46.961 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:41:47.644 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:47.644952 I | ceph-csi: successfully started CSI CephFS driver 2022-04-09T11:41:47.647 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:47.645861 I | op-mon: start running mons 2022-04-09T11:41:47.824 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:47.824780 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:41:47.874 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:47.874395 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-09T11:41:47.880 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:47.880518 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-04-09T11:41:47.880 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:47.880550 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-09T11:41:47.894 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:47.894642 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-04-09T11:41:47.894 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:47.894678 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:41:47.966 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:41:47.987 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-09T11:41:47.988 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:41:48.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:48.250534 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:41:48.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:48.250874 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:41:49.850 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:49.850664 I | op-mon: targeting the mon count 1 2022-04-09T11:41:49.855 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:49.856025 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:41:50.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:50.207615 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:41:50.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:50.207636 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:41:50.542 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:50.542747 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:41:50.543 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:50.542791 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:41:50.877 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:50.877798 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:41:50.878 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:50.877822 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:41:51.269 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:51.269383 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:41:51.269 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:51.269413 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:41:51.601 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:51.602065 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:41:51.602 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:51.602095 I | op-mon: checking for basic quorum with existing mons 2022-04-09T11:41:51.773 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:51.773534 I | op-mon: mon "a" ip is 10.97.79.62 2022-04-09T11:41:51.799 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:51.799821 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:41:51.866 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:51.866200 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:41:51.866 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:51.866493 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:41:52.263 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:52.263243 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-09T11:41:52.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:52.272879 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-09T11:41:52.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:52.272903 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:41:52.455 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:52.453913 I | op-mon: mons running: [a] 2022-04-09T11:41:52.871 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:52.872233 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:41:52.872 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:52.872262 I | op-mon: mons created: 1 2022-04-09T11:41:53.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:53.272959 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:41:53.278 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:53.278545 I | op-mon: mons running: [a] 2022-04-09T11:41:53.684 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:53.684827 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:41:53.685 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:53.684866 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:41:54.111 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:54.111666 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:41:54.531 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:54.531469 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:41:54.983 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:54.984010 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:41:55.426 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:55.426823 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:41:55.427 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:55.426848 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:41:55.851 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:55.851254 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:41:55.851 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:55.851287 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:41:55.852 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:55.851297 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:41:56.316 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:56.316390 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:41:56.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:56.363646 I | op-mgr: start running mgr 2022-04-09T11:41:56.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:56.363674 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:41:56.832 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:56.832797 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-09T11:41:56.843 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:56.843849 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-09T11:41:56.847 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:56.847138 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:41:57.403 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:57.403983 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-04-09T11:41:57.404 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:57.404079 I | op-mgr: successful modules: balancer 2022-04-09T11:41:57.416 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:57.415200 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:41:57.416 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:57.415236 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:41:57.466 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:57.466713 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:41:57.601 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:57.602125 I | op-osd: creating 1 new PVCs for device set "smithi052-lv-1" 2022-04-09T11:41:57.805 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:57.805616 I | op-osd: successfully provisioned PVC "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:41:57.805 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:57.805648 I | op-osd: creating 1 new PVCs for device set "smithi052-lv-2" 2022-04-09T11:41:57.989 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:41:58.026 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.027108 I | op-osd: successfully provisioned PVC "smithi052-lv-2-data-0mh5cw" 2022-04-09T11:41:58.027 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.027206 I | op-osd: creating 1 new PVCs for device set "smithi052-lv-3" 2022-04-09T11:41:58.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.207817 I | op-osd: successfully provisioned PVC "smithi052-lv-3-data-0vswhb" 2022-04-09T11:41:58.208 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.207869 I | op-osd: creating 1 new PVCs for device set "smithi052-lv-4" 2022-04-09T11:41:58.283 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.283636 I | op-mgr: successful modules: dashboard 2022-04-09T11:41:58.286 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.285497 I | op-mgr: successful modules: prometheus 2022-04-09T11:41:58.286 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.286510 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:41:58.407 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.407412 I | op-osd: successfully provisioned PVC "smithi052-lv-4-data-0fj58k" 2022-04-09T11:41:58.419 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.419757 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:41:58.536 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:41:58.555 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-09T11:41:58.555 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:41:58.838 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.838327 I | op-osd: started OSD provisioning job for PVC "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:41:58.838 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:58.838375 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-2-data-0mh5cw" 2022-04-09T11:41:59.284 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:59.284497 I | op-osd: started OSD provisioning job for PVC "smithi052-lv-2-data-0mh5cw" 2022-04-09T11:41:59.284 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:59.284524 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-3-data-0vswhb" 2022-04-09T11:41:59.616 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:59.616895 I | op-osd: started OSD provisioning job for PVC "smithi052-lv-3-data-0vswhb" 2022-04-09T11:41:59.616 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:59.616924 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-4-data-0fj58k" 2022-04-09T11:41:59.999 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:41:59.867246 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:42:00.422 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:00.420599 I | op-osd: started OSD provisioning job for PVC "smithi052-lv-4-data-0fj58k" 2022-04-09T11:42:00.422 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:00.420639 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:42:00.422 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:00.420647 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:42:00.598 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:00.597526 I | op-osd: OSD orchestration status for PVC smithi052-lv-1-data-0z6zxn is "starting" 2022-04-09T11:42:00.599 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:00.597595 I | op-osd: OSD orchestration status for PVC smithi052-lv-2-data-0mh5cw is "starting" 2022-04-09T11:42:00.599 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:00.597629 I | op-osd: OSD orchestration status for PVC smithi052-lv-3-data-0vswhb is "starting" 2022-04-09T11:42:00.599 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:00.597656 I | op-osd: OSD orchestration status for PVC smithi052-lv-4-data-0fj58k is "starting" 2022-04-09T11:42:08.557 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:42:09.383 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:42:09.457 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-09T11:42:09.458 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:42:14.634 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:14.634717 I | op-osd: OSD orchestration status for node smithi052-lv-2-data-0mh5cw is "orchestrating" 2022-04-09T11:42:14.674 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:14.674705 I | op-osd: OSD orchestration status for node smithi052-lv-3-data-0vswhb is "orchestrating" 2022-04-09T11:42:15.166 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:15.166824 I | op-osd: OSD orchestration status for PVC smithi052-lv-2-data-0mh5cw is "orchestrating" 2022-04-09T11:42:15.167 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:15.166872 I | op-osd: OSD orchestration status for PVC smithi052-lv-3-data-0vswhb is "orchestrating" 2022-04-09T11:42:19.459 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:42:20.083 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:42:20.102 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":5,"num_osds":2,"num_up_osds":0,"osd_up_since":0,"num_in_osds":2,"osd_in_since":1649504536,"num_remapped_pgs":0} 2022-04-09T11:42:20.102 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:42:20.745 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:20.746370 I | op-osd: OSD orchestration status for node smithi052-lv-4-data-0fj58k is "orchestrating" 2022-04-09T11:42:21.525 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:21.526002 I | op-osd: OSD orchestration status for PVC smithi052-lv-4-data-0fj58k is "orchestrating" 2022-04-09T11:42:21.530 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:21.527947 I | op-osd: OSD orchestration status for PVC smithi052-lv-3-data-0vswhb is "completed" 2022-04-09T11:42:21.530 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:21.528028 I | op-osd: creating OSD 1 on PVC "smithi052-lv-3-data-0vswhb" 2022-04-09T11:42:21.530 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:21.528063 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-3-data-0vswhb" 2022-04-09T11:42:23.051 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:23.050505 I | op-osd: OSD orchestration status for PVC smithi052-lv-2-data-0mh5cw is "completed" 2022-04-09T11:42:23.051 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:23.050716 I | op-osd: creating OSD 0 on PVC "smithi052-lv-2-data-0mh5cw" 2022-04-09T11:42:23.051 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:23.050839 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-2-data-0mh5cw" 2022-04-09T11:42:23.900 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:23.896947 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi052": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:42:24.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:24.447118 I | op-osd: OSD orchestration status for node smithi052-lv-1-data-0z6zxn is "orchestrating" 2022-04-09T11:42:25.567 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:25.566383 I | op-osd: OSD orchestration status for PVC smithi052-lv-1-data-0z6zxn is "orchestrating" 2022-04-09T11:42:28.511 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:28.511665 I | op-osd: OSD orchestration status for PVC smithi052-lv-4-data-0fj58k is "completed" 2022-04-09T11:42:28.511 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:28.511691 I | op-osd: creating OSD 2 on PVC "smithi052-lv-4-data-0fj58k" 2022-04-09T11:42:28.512 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:28.511699 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-4-data-0fj58k" 2022-04-09T11:42:30.103 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:42:31.642 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:42:31.660 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":7,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1649504547,"num_remapped_pgs":0} 2022-04-09T11:42:31.661 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:42:33.201 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:33.202290 I | op-osd: OSD orchestration status for PVC smithi052-lv-1-data-0z6zxn is "completed" 2022-04-09T11:42:33.202 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:33.202317 I | op-osd: creating OSD 3 on PVC "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:42:33.202 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:33.202325 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:42:35.668 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:35.669136 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-09T11:42:35.668 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:42:35.669154 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:42:41.662 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:42:42.222 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:42:42.239 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":7,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1649504547,"num_remapped_pgs":0} 2022-04-09T11:42:42.239 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:42:52.241 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:42:53.058 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:42:53.078 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":7,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1649504547,"num_remapped_pgs":0} 2022-04-09T11:42:53.078 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:43:03.079 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:43:03.691 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:03.714 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":8,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1649504547,"num_remapped_pgs":0} 2022-04-09T11:43:03.715 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:43:13.716 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd stat -f json 2022-04-09T11:43:14.250 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:14.266 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1649504590,"num_in_osds":4,"osd_in_since":1649504547,"num_remapped_pgs":0} 2022-04-09T11:43:14.267 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-04-09T11:43:14.267 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- cat /etc/ceph/ceph.conf 2022-04-09T11:43:14.451 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2022-04-09T11:43:14.452 INFO:teuthology.orchestra.run.smithi052.stdout:mon_host = 10.97.79.62:6789 2022-04-09T11:43:14.452 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:14.452 INFO:teuthology.orchestra.run.smithi052.stdout:[client.admin] 2022-04-09T11:43:14.452 INFO:teuthology.orchestra.run.smithi052.stdout:keyring = /etc/ceph/keyring 2022-04-09T11:43:14.455 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- cat /etc/ceph/keyring 2022-04-09T11:43:14.631 INFO:teuthology.orchestra.run.smithi052.stdout:[client.admin] 2022-04-09T11:43:14.631 INFO:teuthology.orchestra.run.smithi052.stdout:key = AQCgcFFiV5qJLxAARJLRguz3k/KriV+7LCAVmA== 2022-04-09T11:43:14.636 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph 2022-04-09T11:43:14.646 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:43:14.647 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.conf 2022-04-09T11:43:14.660 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:43:14.661 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/keyring 2022-04-09T11:43:14.679 INFO:tasks.rook:Setting up client nodes... 2022-04-09T11:43:14.679 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-04-09T11:43:15.392 INFO:teuthology.orchestra.run.smithi052.stdout:[client.a] 2022-04-09T11:43:15.392 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQBTcVFiuHIsFBAAxnJVIQtt93B88L1oISuB8w== 2022-04-09T11:43:15.412 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2022-04-09T11:43:15.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-04-09T11:43:15.413 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-04-09T11:43:15.432 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd dump --format=json 2022-04-09T11:43:15.973 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:15.973 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"fsid":"0ebce0d0-fd08-4b7b-9d0f-4c750a9769fa","created":"2022-04-09T11:40:42.390384+0000","modified":"2022-04-09T11:43:12.749187+0000","last_up_change":"2022-04-09T11:43:10.421397+0000","last_in_change":"2022-04-09T11:42:27.671284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-09T11:43:10.361169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"012307a7-4f35-421c-8a63-1dd236bb9adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6800","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6801","nonce":2706377495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6802","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6803","nonce":2706377495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6806","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6807","nonce":2706377495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6804","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6805","nonce":2706377495}]},"public_addr":"10.249.152.22:6801/2706377495","cluster_addr":"10.249.152.22:6803/2706377495","heartbeat_back_addr":"10.249.152.22:6807/2706377495","heartbeat_front_addr":"10.249.152.22:6805/2706377495","state":["exists","up"]},{"osd":1,"uuid":"791145dd-4eb8-46e4-97fb-5e563911099b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6800","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6801","nonce":4071012033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6802","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6803","nonce":4071012033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6806","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6807","nonce":4071012033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6804","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6805","nonce":4071012033}]},"public_addr":"10.249.152.21:6801/4071012033","cluster_addr":"10.249.152.21:6803/4071012033","heartbeat_back_addr":"10.249.152.21:6807/4071012033","heartbeat_front_addr":"10.249.152.21:6805/4071012033","state":["exists","up"]},{"osd":2,"uuid":"a1fa3501-f8c9-4be0-9f8f-0e2c2cbde103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6800","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6801","nonce":59160399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6802","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6803","nonce":59160399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6806","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6807","nonce":59160399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6804","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6805","nonce":59160399}]},"public_addr":"10.249.152.23:6801/59160399","cluster_addr":"10.249.152.23:6803/59160399","heartbeat_back_addr":"10.249.152.23:6807/59160399","heartbeat_front_addr":"10.249.152.23:6805/59160399","state":["exists","up"]},{"osd":3,"uuid":"5dd569c2-584c-425e-ab39-123e8bd31927","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6800","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6801","nonce":1418324677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6802","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6803","nonce":1418324677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6806","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6807","nonce":1418324677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6804","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6805","nonce":1418324677}]},"public_addr":"10.249.152.24:6801/1418324677","cluster_addr":"10.249.152.24:6803/1418324677","heartbeat_back_addr":"10.249.152.24:6807/1418324677","heartbeat_front_addr":"10.249.152.24:6805/1418324677","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:06.556953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:03.085633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:03.217146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:08.208775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.152.12:0/1950373753":"2022-04-10T11:41:26.281013+0000","10.249.152.12:0/3134960114":"2022-04-10T11:41:26.281013+0000","10.249.152.12:0/2903658055":"2022-04-10T11:41:26.281013+0000","10.249.152.12:6801/1973147197":"2022-04-10T11:41:26.281013+0000","10.249.152.12:6800/1973147197":"2022-04-10T11:41:26.281013+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-09T11:43:15.995 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-04-09T11:43:10.361169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-04-09T11:43:15.996 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd pool get .mgr pg_num 2022-04-09T11:43:16.529 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2022-04-09T11:43:16.552 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-04-09T11:43:16.553 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-04-09T11:43:16.553 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph mgr dump --format=json 2022-04-09T11:43:17.116 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:17.134 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":14,"active_gid":4162,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.12:6800","nonce":286832564},{"type":"v1","addr":"10.249.152.12:6801","nonce":286832564}]},"active_addr":"10.249.152.12:6801/286832564","active_change":"2022-04-09T11:41:33.392242+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.249.152.12:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.249.152.12:0","nonce":362723292}]},{"addrvec":[{"type":"v2","addr":"10.249.152.12:0","nonce":476372656}]},{"addrvec":[{"type":"v2","addr":"10.249.152.12:0","nonce":4202881964}]},{"addrvec":[{"type":"v2","addr":"10.249.152.12:0","nonce":3302254343}]}]}} 2022-04-09T11:43:17.139 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-04-09T11:43:17.139 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-04-09T11:43:17.140 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd dump --format=json 2022-04-09T11:43:17.694 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:17.694 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"fsid":"0ebce0d0-fd08-4b7b-9d0f-4c750a9769fa","created":"2022-04-09T11:40:42.390384+0000","modified":"2022-04-09T11:43:12.749187+0000","last_up_change":"2022-04-09T11:43:10.421397+0000","last_in_change":"2022-04-09T11:42:27.671284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-09T11:43:10.361169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"012307a7-4f35-421c-8a63-1dd236bb9adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6800","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6801","nonce":2706377495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6802","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6803","nonce":2706377495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6806","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6807","nonce":2706377495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6804","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6805","nonce":2706377495}]},"public_addr":"10.249.152.22:6801/2706377495","cluster_addr":"10.249.152.22:6803/2706377495","heartbeat_back_addr":"10.249.152.22:6807/2706377495","heartbeat_front_addr":"10.249.152.22:6805/2706377495","state":["exists","up"]},{"osd":1,"uuid":"791145dd-4eb8-46e4-97fb-5e563911099b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6800","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6801","nonce":4071012033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6802","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6803","nonce":4071012033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6806","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6807","nonce":4071012033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6804","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6805","nonce":4071012033}]},"public_addr":"10.249.152.21:6801/4071012033","cluster_addr":"10.249.152.21:6803/4071012033","heartbeat_back_addr":"10.249.152.21:6807/4071012033","heartbeat_front_addr":"10.249.152.21:6805/4071012033","state":["exists","up"]},{"osd":2,"uuid":"a1fa3501-f8c9-4be0-9f8f-0e2c2cbde103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6800","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6801","nonce":59160399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6802","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6803","nonce":59160399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6806","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6807","nonce":59160399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6804","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6805","nonce":59160399}]},"public_addr":"10.249.152.23:6801/59160399","cluster_addr":"10.249.152.23:6803/59160399","heartbeat_back_addr":"10.249.152.23:6807/59160399","heartbeat_front_addr":"10.249.152.23:6805/59160399","state":["exists","up"]},{"osd":3,"uuid":"5dd569c2-584c-425e-ab39-123e8bd31927","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6800","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6801","nonce":1418324677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6802","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6803","nonce":1418324677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6806","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6807","nonce":1418324677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6804","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6805","nonce":1418324677}]},"public_addr":"10.249.152.24:6801/1418324677","cluster_addr":"10.249.152.24:6803/1418324677","heartbeat_back_addr":"10.249.152.24:6807/1418324677","heartbeat_front_addr":"10.249.152.24:6805/1418324677","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:06.556953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:03.085633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:03.217146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:08.208775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.152.12:0/1950373753":"2022-04-10T11:41:26.281013+0000","10.249.152.12:0/3134960114":"2022-04-10T11:41:26.281013+0000","10.249.152.12:0/2903658055":"2022-04-10T11:41:26.281013+0000","10.249.152.12:6801/1973147197":"2022-04-10T11:41:26.281013+0000","10.249.152.12:6800/1973147197":"2022-04-10T11:41:26.281013+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-09T11:43:17.715 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-04-09T11:43:17.716 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd dump --format=json 2022-04-09T11:43:18.237 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:18.237 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"fsid":"0ebce0d0-fd08-4b7b-9d0f-4c750a9769fa","created":"2022-04-09T11:40:42.390384+0000","modified":"2022-04-09T11:43:12.749187+0000","last_up_change":"2022-04-09T11:43:10.421397+0000","last_in_change":"2022-04-09T11:42:27.671284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-09T11:43:10.361169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"012307a7-4f35-421c-8a63-1dd236bb9adc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6800","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6801","nonce":2706377495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6802","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6803","nonce":2706377495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6806","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6807","nonce":2706377495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.22:6804","nonce":2706377495},{"type":"v1","addr":"10.249.152.22:6805","nonce":2706377495}]},"public_addr":"10.249.152.22:6801/2706377495","cluster_addr":"10.249.152.22:6803/2706377495","heartbeat_back_addr":"10.249.152.22:6807/2706377495","heartbeat_front_addr":"10.249.152.22:6805/2706377495","state":["exists","up"]},{"osd":1,"uuid":"791145dd-4eb8-46e4-97fb-5e563911099b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6800","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6801","nonce":4071012033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6802","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6803","nonce":4071012033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6806","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6807","nonce":4071012033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.21:6804","nonce":4071012033},{"type":"v1","addr":"10.249.152.21:6805","nonce":4071012033}]},"public_addr":"10.249.152.21:6801/4071012033","cluster_addr":"10.249.152.21:6803/4071012033","heartbeat_back_addr":"10.249.152.21:6807/4071012033","heartbeat_front_addr":"10.249.152.21:6805/4071012033","state":["exists","up"]},{"osd":2,"uuid":"a1fa3501-f8c9-4be0-9f8f-0e2c2cbde103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6800","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6801","nonce":59160399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6802","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6803","nonce":59160399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6806","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6807","nonce":59160399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.23:6804","nonce":59160399},{"type":"v1","addr":"10.249.152.23:6805","nonce":59160399}]},"public_addr":"10.249.152.23:6801/59160399","cluster_addr":"10.249.152.23:6803/59160399","heartbeat_back_addr":"10.249.152.23:6807/59160399","heartbeat_front_addr":"10.249.152.23:6805/59160399","state":["exists","up"]},{"osd":3,"uuid":"5dd569c2-584c-425e-ab39-123e8bd31927","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6800","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6801","nonce":1418324677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6802","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6803","nonce":1418324677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6806","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6807","nonce":1418324677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.249.152.24:6804","nonce":1418324677},{"type":"v1","addr":"10.249.152.24:6805","nonce":1418324677}]},"public_addr":"10.249.152.24:6801/1418324677","cluster_addr":"10.249.152.24:6803/1418324677","heartbeat_back_addr":"10.249.152.24:6807/1418324677","heartbeat_front_addr":"10.249.152.24:6805/1418324677","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:06.556953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:03.085633+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:03.217146+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:43:08.208775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.249.152.12:0/1950373753":"2022-04-10T11:41:26.281013+0000","10.249.152.12:0/3134960114":"2022-04-10T11:41:26.281013+0000","10.249.152.12:0/2903658055":"2022-04-10T11:41:26.281013+0000","10.249.152.12:6801/1973147197":"2022-04-10T11:41:26.281013+0000","10.249.152.12:6800/1973147197":"2022-04-10T11:41:26.281013+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-09T11:43:18.259 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph tell osd.0 flush_pg_stats 2022-04-09T11:43:18.259 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph tell osd.1 flush_pg_stats 2022-04-09T11:43:18.260 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph tell osd.2 flush_pg_stats 2022-04-09T11:43:18.261 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph tell osd.3 flush_pg_stats 2022-04-09T11:43:18.617 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2022-04-09T11:43:18.618 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd last-stat-seq osd.1 2022-04-09T11:43:18.639 INFO:teuthology.orchestra.run.smithi052.stdout:47244640262 2022-04-09T11:43:18.639 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd last-stat-seq osd.2 2022-04-09T11:43:18.730 INFO:teuthology.orchestra.run.smithi052.stdout:64424509444 2022-04-09T11:43:18.731 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd last-stat-seq osd.3 2022-04-09T11:43:18.811 INFO:teuthology.orchestra.run.smithi052.stdout:55834574852 2022-04-09T11:43:18.811 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd last-stat-seq osd.0 2022-04-09T11:43:19.199 INFO:teuthology.orchestra.run.smithi052.stdout:42949672963 2022-04-09T11:43:19.220 INFO:tasks.rook.ceph_manager.ceph:need seq 42949672964 got 42949672963 for osd.1 2022-04-09T11:43:19.295 INFO:teuthology.orchestra.run.smithi052.stdout:64424509443 2022-04-09T11:43:19.316 INFO:tasks.rook.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.3 2022-04-09T11:43:19.357 INFO:teuthology.orchestra.run.smithi052.stdout:47244640261 2022-04-09T11:43:19.379 INFO:tasks.rook.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.2 2022-04-09T11:43:19.430 INFO:teuthology.orchestra.run.smithi052.stdout:55834574851 2022-04-09T11:43:19.449 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2022-04-09T11:43:20.222 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd last-stat-seq osd.1 2022-04-09T11:43:20.317 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd last-stat-seq osd.3 2022-04-09T11:43:20.381 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd last-stat-seq osd.2 2022-04-09T11:43:20.451 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph osd last-stat-seq osd.0 2022-04-09T11:43:20.789 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2022-04-09T11:43:20.814 INFO:tasks.rook.ceph_manager.ceph:need seq 42949672964 got 42949672964 for osd.1 2022-04-09T11:43:20.814 DEBUG:teuthology.parallel:result is None 2022-04-09T11:43:20.888 INFO:teuthology.orchestra.run.smithi052.stdout:64424509444 2022-04-09T11:43:20.910 INFO:tasks.rook.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.3 2022-04-09T11:43:20.910 DEBUG:teuthology.parallel:result is None 2022-04-09T11:43:20.921 INFO:teuthology.orchestra.run.smithi052.stdout:47244640262 2022-04-09T11:43:20.939 INFO:tasks.rook.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.2 2022-04-09T11:43:20.940 DEBUG:teuthology.parallel:result is None 2022-04-09T11:43:20.999 INFO:teuthology.orchestra.run.smithi052.stdout:55834574853 2022-04-09T11:43:21.019 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.0 2022-04-09T11:43:21.019 DEBUG:teuthology.parallel:result is None 2022-04-09T11:43:21.020 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-04-09T11:43:21.020 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph pg dump --format=json 2022-04-09T11:43:21.511 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:21.514 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2022-04-09T11:43:21.530 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2022-04-09T11:43:19.848919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23488,"kb_used_data":1956,"kb_used_omap":0,"kb_used_meta":20672,"kb_avail":374957120,"statfs":{"total":383980142592,"available":383956090880,"internally_reserved":0,"allocated":2002944,"data_stored":1573864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21168128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000617"},"pg_stats":[{"pgid":"1.0","version":"17'97","reported_seq":137,"reported_epoch":17,"state":"active+clean","last_fresh":"2022-04-09T11:43:16.661241+0000","last_change":"2022-04-09T11:43:12.522517+0000","last_active":"2022-04-09T11:43:16.661241+0000","last_peered":"2022-04-09T11:43:16.661241+0000","last_clean":"2022-04-09T11:43:16.661241+0000","last_became_active":"2022-04-09T11:43:12.520996+0000","last_became_peered":"2022-04-09T11:43:12.520996+0000","last_unstale":"2022-04-09T11:43:16.661241+0000","last_undegraded":"2022-04-09T11:43:16.661241+0000","last_fullsized":"2022-04-09T11:43:16.661241+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-09T11:43:10.421397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-09T11:43:10.421397+0000","last_clean_scrub_stamp":"2022-04-09T11:43:10.421397+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-10T12:43:06.895715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5016,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740136,"statfs":{"total":95995035648,"available":95989899264,"internally_reserved":0,"allocated":147456,"data_stored":43249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":618496,"data_stored":510205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]}]},{"osd":1,"up_from":10,"seq":42949672964,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6308,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738844,"statfs":{"total":95995035648,"available":95988576256,"internally_reserved":0,"allocated":618496,"data_stored":510205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5860,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739292,"statfs":{"total":95995035648,"available":95989035008,"internally_reserved":0,"allocated":618496,"data_stored":510205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-09T11:43:21.531 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph pg dump --format=json 2022-04-09T11:43:22.064 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:22.067 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2022-04-09T11:43:22.086 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2022-04-09T11:43:21.849480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23488,"kb_used_data":1956,"kb_used_omap":0,"kb_used_meta":20672,"kb_avail":374957120,"statfs":{"total":383980142592,"available":383956090880,"internally_reserved":0,"allocated":2002944,"data_stored":1573864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21168128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001178"},"pg_stats":[{"pgid":"1.0","version":"17'97","reported_seq":137,"reported_epoch":17,"state":"active+clean","last_fresh":"2022-04-09T11:43:16.661241+0000","last_change":"2022-04-09T11:43:12.522517+0000","last_active":"2022-04-09T11:43:16.661241+0000","last_peered":"2022-04-09T11:43:16.661241+0000","last_clean":"2022-04-09T11:43:16.661241+0000","last_became_active":"2022-04-09T11:43:12.520996+0000","last_became_peered":"2022-04-09T11:43:12.520996+0000","last_unstale":"2022-04-09T11:43:16.661241+0000","last_undegraded":"2022-04-09T11:43:16.661241+0000","last_fullsized":"2022-04-09T11:43:16.661241+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-09T11:43:10.421397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-09T11:43:10.421397+0000","last_clean_scrub_stamp":"2022-04-09T11:43:10.421397+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-10T12:43:06.895715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5016,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740136,"statfs":{"total":95995035648,"available":95989899264,"internally_reserved":0,"allocated":147456,"data_stored":43249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":618496,"data_stored":510205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]}]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6308,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738844,"statfs":{"total":95995035648,"available":95988576256,"internally_reserved":0,"allocated":618496,"data_stored":510205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5860,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739292,"statfs":{"total":95995035648,"available":95989035008,"internally_reserved":0,"allocated":618496,"data_stored":510205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-09T11:43:22.086 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-04-09T11:43:22.087 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-04-09T11:43:22.087 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-04-09T11:43:22.088 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph health --format=json 2022-04-09T11:43:22.624 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:22.624 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-04-09T11:43:22.643 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-04-09T11:43:22.643 INFO:tasks.rook:Rook complete, yielding 2022-04-09T11:43:22.643 INFO:teuthology.run_tasks:Running task rook.shell... 2022-04-09T11:43:22.656 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph -s 2022-04-09T11:43:23.255 INFO:teuthology.orchestra.run.smithi052.stdout: cluster: 2022-04-09T11:43:23.256 INFO:teuthology.orchestra.run.smithi052.stdout: id: 0ebce0d0-fd08-4b7b-9d0f-4c750a9769fa 2022-04-09T11:43:23.256 INFO:teuthology.orchestra.run.smithi052.stdout: health: HEALTH_OK 2022-04-09T11:43:23.256 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:23.256 INFO:teuthology.orchestra.run.smithi052.stdout: services: 2022-04-09T11:43:23.257 INFO:teuthology.orchestra.run.smithi052.stdout: mon: 1 daemons, quorum a (age 2m) 2022-04-09T11:43:23.257 INFO:teuthology.orchestra.run.smithi052.stdout: mgr: a(active, since 109s) 2022-04-09T11:43:23.257 INFO:teuthology.orchestra.run.smithi052.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 55s) 2022-04-09T11:43:23.257 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:23.257 INFO:teuthology.orchestra.run.smithi052.stdout: data: 2022-04-09T11:43:23.258 INFO:teuthology.orchestra.run.smithi052.stdout: pools: 1 pools, 1 pgs 2022-04-09T11:43:23.258 INFO:teuthology.orchestra.run.smithi052.stdout: objects: 2 objects, 577 KiB 2022-04-09T11:43:23.258 INFO:teuthology.orchestra.run.smithi052.stdout: usage: 23 MiB used, 358 GiB / 358 GiB avail 2022-04-09T11:43:23.258 INFO:teuthology.orchestra.run.smithi052.stdout: pgs: 1 active+clean 2022-04-09T11:43:23.259 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:23.259 INFO:teuthology.orchestra.run.smithi052.stdout: io: 2022-04-09T11:43:23.259 INFO:teuthology.orchestra.run.smithi052.stdout: client: 2.2 KiB/s rd, 65 KiB/s wr, 2 op/s rd, 3 op/s wr 2022-04-09T11:43:23.259 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:43:23.279 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch status 2022-04-09T11:43:23.943 INFO:teuthology.orchestra.run.smithi052.stdout:Backend: rook 2022-04-09T11:43:23.944 INFO:teuthology.orchestra.run.smithi052.stdout:Available: Yes 2022-04-09T11:43:23.968 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ps 2022-04-09T11:43:24.539 INFO:teuthology.orchestra.run.smithi052.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-04-09T11:43:24.539 INFO:teuthology.orchestra.run.smithi052.stdout:crashcollector.crash smithi052 running (61s) 0s ago 61s - - 2abf022dd4de 2022-04-09T11:43:24.540 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.a smithi052 running (2m) 0s ago 2m - - 2abf022dd4de 2022-04-09T11:43:24.540 INFO:teuthology.orchestra.run.smithi052.stdout:mon.a smithi052 running (3m) 0s ago 3m - - 2abf022dd4de 2022-04-09T11:43:24.540 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 smithi052 running (60s) 0s ago 60s - - 2abf022dd4de 2022-04-09T11:43:24.541 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 smithi052 running (61s) 0s ago 61s - - 2abf022dd4de 2022-04-09T11:43:24.541 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 smithi052 running (55s) 0s ago 55s - - 2abf022dd4de 2022-04-09T11:43:24.541 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 smithi052 running (50s) 0s ago 50s - - 2abf022dd4de 2022-04-09T11:43:24.562 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls 2022-04-09T11:43:25.186 INFO:teuthology.orchestra.run.smithi052.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-04-09T11:43:25.187 INFO:teuthology.orchestra.run.smithi052.stdout:crash 1/1 0s ago 62s * 2022-04-09T11:43:25.187 INFO:teuthology.orchestra.run.smithi052.stdout:mgr 1/1 0s ago 2m count:1 2022-04-09T11:43:25.187 INFO:teuthology.orchestra.run.smithi052.stdout:mon 1/1 0s ago 3m count:1 2022-04-09T11:43:25.188 INFO:teuthology.orchestra.run.smithi052.stdout:osd 8 0s ago 62s 2022-04-09T11:43:25.188 INFO:teuthology.orchestra.run.smithi052.stdout:osd.all-available-devices 0 0s ago - * 2022-04-09T11:43:25.216 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch device ls 2022-04-09T11:43:25.767 INFO:teuthology.orchestra.run.smithi052.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-04-09T11:43:25.767 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-09T11:43:25.767 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-09T11:43:25.767 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-09T11:43:25.768 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-09T11:43:25.789 INFO:teuthology.run_tasks:Running task exec... 2022-04-09T11:43:25.802 INFO:teuthology.task.exec:Executing custom commands... 2022-04-09T11:43:25.802 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi052.front.sepia.ceph.com 2022-04-09T11:43:25.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-04-09T11:43:25.803 DEBUG:teuthology.orchestra.run.smithi052:> toolbox() { 2022-04-09T11:43:25.803 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-04-09T11:43:25.804 DEBUG:teuthology.orchestra.run.smithi052:> } 2022-04-09T11:43:25.804 DEBUG:teuthology.orchestra.run.smithi052:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-04-09T11:43:25.804 DEBUG:teuthology.orchestra.run.smithi052:> toolbox ceph orch osd rm 0 --force 2022-04-09T11:43:25.804 DEBUG:teuthology.orchestra.run.smithi052:> removed_pv="" 2022-04-09T11:43:25.805 DEBUG:teuthology.orchestra.run.smithi052:> while [ "$removed_pv" = "" ] 2022-04-09T11:43:25.805 DEBUG:teuthology.orchestra.run.smithi052:> do 2022-04-09T11:43:25.805 DEBUG:teuthology.orchestra.run.smithi052:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-04-09T11:43:25.806 DEBUG:teuthology.orchestra.run.smithi052:> sleep 3s 2022-04-09T11:43:25.806 DEBUG:teuthology.orchestra.run.smithi052:> done 2022-04-09T11:43:25.806 DEBUG:teuthology.orchestra.run.smithi052:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-04-09T11:43:25.807 DEBUG:teuthology.orchestra.run.smithi052:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-04-09T11:43:25.807 DEBUG:teuthology.orchestra.run.smithi052:> toolbox ceph orch device zap $host $target_path --force 2022-04-09T11:43:25.807 DEBUG:teuthology.orchestra.run.smithi052:> zap_completion="0" 2022-04-09T11:43:25.807 DEBUG:teuthology.orchestra.run.smithi052:> while [ "$zap_completion" = "0" ] 2022-04-09T11:43:25.808 DEBUG:teuthology.orchestra.run.smithi052:> do 2022-04-09T11:43:25.808 DEBUG:teuthology.orchestra.run.smithi052:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-04-09T11:43:25.808 DEBUG:teuthology.orchestra.run.smithi052:> sleep 3s 2022-04-09T11:43:25.808 DEBUG:teuthology.orchestra.run.smithi052:> done 2022-04-09T11:43:25.809 DEBUG:teuthology.orchestra.run.smithi052:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-04-09T11:43:25.809 DEBUG:teuthology.orchestra.run.smithi052:> toolbox ceph orch apply osd --all-available-devices 2022-04-09T11:43:25.809 DEBUG:teuthology.orchestra.run.smithi052:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-04-09T11:43:25.810 DEBUG:teuthology.orchestra.run.smithi052:> num_osd="0" 2022-04-09T11:43:25.810 DEBUG:teuthology.orchestra.run.smithi052:> while [ "$num_osd" != "$orig_num_osd" ] 2022-04-09T11:43:25.810 DEBUG:teuthology.orchestra.run.smithi052:> do 2022-04-09T11:43:25.810 DEBUG:teuthology.orchestra.run.smithi052:> echo "waiting for osd to come back up" 2022-04-09T11:43:25.811 DEBUG:teuthology.orchestra.run.smithi052:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-04-09T11:43:25.811 DEBUG:teuthology.orchestra.run.smithi052:> sleep 30s 2022-04-09T11:43:25.811 DEBUG:teuthology.orchestra.run.smithi052:> done 2022-04-09T11:43:25.812 DEBUG:teuthology.orchestra.run.smithi052:> ' 2022-04-09T11:43:25.829 INFO:teuthology.orchestra.run.smithi052.stderr:++ toolbox ceph osd stat 2022-04-09T11:43:25.829 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:43:25.830 INFO:teuthology.orchestra.run.smithi052.stderr:++ cut -f3 -d ' ' 2022-04-09T11:43:27.774 INFO:teuthology.orchestra.run.smithi052.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:28.272 INFO:teuthology.orchestra.run.smithi052.stderr:+ orig_num_osd=4 2022-04-09T11:43:28.272 INFO:teuthology.orchestra.run.smithi052.stderr:+ toolbox ceph orch osd rm 0 --force 2022-04-09T11:43:28.272 INFO:teuthology.orchestra.run.smithi052.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-04-09T11:43:28.333 INFO:teuthology.orchestra.run.smithi052.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:28.891 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.889909 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-09T11:43:28.892 INFO:tasks.rook.operator.smithi052.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", AllowUnsupported: true}, 2022-04-09T11:43:28.892 INFO:tasks.rook.operator.smithi052.stdout: Storage: v1.StorageScopeSpec{ 2022-04-09T11:43:28.892 INFO:tasks.rook.operator.smithi052.stdout: ... // 3 identical fields 2022-04-09T11:43:28.892 INFO:tasks.rook.operator.smithi052.stdout: Config: nil, 2022-04-09T11:43:28.893 INFO:tasks.rook.operator.smithi052.stdout: Selection: {}, 2022-04-09T11:43:28.893 INFO:tasks.rook.operator.smithi052.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-04-09T11:43:28.893 INFO:tasks.rook.operator.smithi052.stdout: {Name: "smithi052-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi052-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:28.893 INFO:tasks.rook.operator.smithi052.stdout:- { 2022-04-09T11:43:28.894 INFO:tasks.rook.operator.smithi052.stdout:- Name: "smithi052-lv-2", 2022-04-09T11:43:28.894 INFO:tasks.rook.operator.smithi052.stdout:- Count: 1, 2022-04-09T11:43:28.894 INFO:tasks.rook.operator.smithi052.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-04-09T11:43:28.894 INFO:tasks.rook.operator.smithi052.stdout:- }, 2022-04-09T11:43:28.894 INFO:tasks.rook.operator.smithi052.stdout: {Name: "smithi052-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi052-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:28.895 INFO:tasks.rook.operator.smithi052.stdout: {Name: "smithi052-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi052-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:28.895 INFO:tasks.rook.operator.smithi052.stdout: }, 2022-04-09T11:43:28.895 INFO:tasks.rook.operator.smithi052.stdout: }, 2022-04-09T11:43:28.895 INFO:tasks.rook.operator.smithi052.stdout: Annotations: nil, 2022-04-09T11:43:28.896 INFO:tasks.rook.operator.smithi052.stdout: Labels: nil, 2022-04-09T11:43:28.896 INFO:tasks.rook.operator.smithi052.stdout: ... // 20 identical fields 2022-04-09T11:43:28.896 INFO:tasks.rook.operator.smithi052.stdout: } 2022-04-09T11:43:28.896 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.890040 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-04-09T11:43:28.896 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.890406 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-04-09T11:43:28.897 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.890431 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-04-09T11:43:28.897 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.890439 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-04-09T11:43:28.897 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.890447 I | operator: successfully started the controller-runtime manager 2022-04-09T11:43:28.897 INFO:tasks.rook.operator.smithi052.stdout:I0409 11:43:28.890590 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-04-09T11:43:28.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.922065 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-09T11:43:28.921 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.922097 I | operator: watching all namespaces for Ceph CRs 2022-04-09T11:43:28.921 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.922135 I | operator: setting up schemes 2022-04-09T11:43:28.928 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:28.927221 I | operator: setting up the controller-runtime manager 2022-04-09T11:43:29.128 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.129867 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-09T11:43:29.216 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.217191 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-09T11:43:29.216 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.217247 I | ceph-cluster-controller: successfully started 2022-04-09T11:43:29.217 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.217333 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-09T11:43:29.217 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.217385 I | ceph-crashcollector-controller: successfully started 2022-04-09T11:43:29.218 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.217409 I | ceph-block-pool-controller: successfully started 2022-04-09T11:43:29.218 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.217447 I | ceph-object-store-user-controller: successfully started 2022-04-09T11:43:29.218 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.217476 I | ceph-object-realm-controller: successfully started 2022-04-09T11:43:29.225 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.225888 I | ceph-object-zonegroup-controller: successfully started 2022-04-09T11:43:29.244 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.225936 I | ceph-object-zone-controller: successfully started 2022-04-09T11:43:29.245 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226107 I | ceph-object-controller: successfully started 2022-04-09T11:43:29.245 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226154 I | ceph-file-controller: successfully started 2022-04-09T11:43:29.246 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226200 I | ceph-nfs-controller: successfully started 2022-04-09T11:43:29.246 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226230 I | ceph-rbd-mirror-controller: successfully started 2022-04-09T11:43:29.246 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226253 I | ceph-client-controller: successfully started 2022-04-09T11:43:29.247 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226275 I | ceph-filesystem-mirror-controller: successfully started 2022-04-09T11:43:29.247 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226298 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-09T11:43:29.248 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226317 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-09T11:43:29.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226332 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-09T11:43:29.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226350 I | ceph-bucket-topic: successfully started 2022-04-09T11:43:29.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226361 I | ceph-bucket-notification: successfully started 2022-04-09T11:43:29.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226373 I | ceph-bucket-notification: successfully started 2022-04-09T11:43:29.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226383 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-09T11:43:29.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.226392 I | blockpool-rados-namespace-controller: successfully started 2022-04-09T11:43:29.256 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.247247 I | operator: starting the controller-runtime manager 2022-04-09T11:43:29.355 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.356875 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-09T11:43:29.356 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.356912 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-09T11:43:29.357 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.356922 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-09T11:43:29.357 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.357116 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:43:29.359 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.360418 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-09T11:43:29.432 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.433178 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:43:29.432 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.433252 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:43:29.432 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.433302 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-04-09T11:43:29.433 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.433365 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:43:29.435 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.435997 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-04-09T11:43:29.435 INFO:tasks.rook.operator.smithi052.stdout:I0409 11:43:29.436077 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:43:29.447 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.448413 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-04-09T11:43:29.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.452200 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:43:29.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.452239 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:43:29.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.452251 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:43:29.518 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517586 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-04-09T11:43:29.518 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517612 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-04-09T11:43:29.519 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517617 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-04-09T11:43:29.519 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517622 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-04-09T11:43:29.519 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517627 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-04-09T11:43:29.519 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517632 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.6.0" (default) 2022-04-09T11:43:29.520 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517638 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-04-09T11:43:29.520 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517644 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-04-09T11:43:29.520 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517649 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-04-09T11:43:29.521 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517655 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-04-09T11:43:29.521 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517660 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-04-09T11:43:29.521 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517665 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-04-09T11:43:29.521 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517670 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-04-09T11:43:29.522 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517674 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-04-09T11:43:29.522 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517678 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-04-09T11:43:29.522 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517683 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.6.0" 2022-04-09T11:43:29.522 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517752 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:43:29.523 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.517800 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:43:29.523 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:29.519531 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:43:33.132 INFO:teuthology.orchestra.run.smithi052.stdout:Success 2022-04-09T11:43:33.156 INFO:teuthology.orchestra.run.smithi052.stderr:+ removed_pv= 2022-04-09T11:43:33.157 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' '' = '' ']' 2022-04-09T11:43:33.157 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl get pv 2022-04-09T11:43:33.157 INFO:teuthology.orchestra.run.smithi052.stderr:++ grep Released 2022-04-09T11:43:33.158 INFO:teuthology.orchestra.run.smithi052.stderr:++ cut -f1 -d ' ' 2022-04-09T11:43:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:+ removed_pv= 2022-04-09T11:43:33.244 INFO:teuthology.orchestra.run.smithi052.stderr:+ sleep 3s 2022-04-09T11:43:36.245 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' '' = '' ']' 2022-04-09T11:43:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl get pv 2022-04-09T11:43:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:++ grep Released 2022-04-09T11:43:36.246 INFO:teuthology.orchestra.run.smithi052.stderr:++ cut -f1 -d ' ' 2022-04-09T11:43:36.374 INFO:teuthology.orchestra.run.smithi052.stderr:+ removed_pv= 2022-04-09T11:43:36.374 INFO:teuthology.orchestra.run.smithi052.stderr:+ sleep 3s 2022-04-09T11:43:37.237 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:37.238609 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-04-09T11:43:39.375 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' '' = '' ']' 2022-04-09T11:43:39.376 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl get pv 2022-04-09T11:43:39.376 INFO:teuthology.orchestra.run.smithi052.stderr:++ grep Released 2022-04-09T11:43:39.376 INFO:teuthology.orchestra.run.smithi052.stderr:++ cut -f1 -d ' ' 2022-04-09T11:43:39.440 INFO:teuthology.orchestra.run.smithi052.stderr:+ removed_pv=smithi052-lv-2 2022-04-09T11:43:39.441 INFO:teuthology.orchestra.run.smithi052.stderr:+ sleep 3s 2022-04-09T11:43:42.442 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' smithi052-lv-2 = '' ']' 2022-04-09T11:43:42.443 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl get pv smithi052-lv-2 -o 'jsonpath={.spec.local.path}' 2022-04-09T11:43:42.787 INFO:teuthology.orchestra.run.smithi052.stderr:+ target_path=/dev/vg_nvme/lv_2 2022-04-09T11:43:42.788 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789372 I | ceph-csi: Detected ceph CSI image version: "v3.6.0" 2022-04-09T11:43:42.788 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789406 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-04-09T11:43:42.789 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789413 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-09T11:43:42.789 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789417 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-09T11:43:42.789 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789422 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-09T11:43:42.789 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789426 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-09T11:43:42.790 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789430 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-09T11:43:42.790 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789432 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-09T11:43:42.790 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789435 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-09T11:43:42.791 INFO:teuthology.orchestra.run.smithi052.stderr:++ cut -f1 -d - 2022-04-09T11:43:42.791 INFO:teuthology.orchestra.run.smithi052.stderr:++ echo smithi052-lv-2 2022-04-09T11:43:42.792 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789437 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-09T11:43:42.792 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789440 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-09T11:43:42.792 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789442 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-09T11:43:42.793 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789445 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:43:42.793 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789448 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:43:42.794 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789452 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-04-09T11:43:42.794 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789457 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-04-09T11:43:42.795 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789462 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-04-09T11:43:42.795 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789465 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-04-09T11:43:42.795 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789468 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-04-09T11:43:42.796 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789471 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:43:42.796 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789473 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:43:42.797 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789476 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-04-09T11:43:42.797 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789484 I | ceph-csi: Kubernetes version is 1.23 2022-04-09T11:43:42.798 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789493 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-04-09T11:43:42.798 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:42.789499 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-04-09T11:43:42.799 INFO:teuthology.orchestra.run.smithi052.stderr:+ host=smithi052 2022-04-09T11:43:42.800 INFO:teuthology.orchestra.run.smithi052.stderr:+ toolbox ceph orch device zap smithi052 /dev/vg_nvme/lv_2 --force 2022-04-09T11:43:42.800 INFO:teuthology.orchestra.run.smithi052.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi052 /dev/vg_nvme/lv_2 --force 2022-04-09T11:43:43.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.273000 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:43:43.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.273051 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:43:43.273 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.273059 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:43:43.273 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.273064 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:43:43.273 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.273068 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:43:43.273 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.273072 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:43:43.274 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.273077 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-04-09T11:43:43.446 INFO:teuthology.orchestra.run.smithi052.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:43.466 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.467301 I | ceph-spec: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:43:43.467 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.467333 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:43:43.636 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.635834 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:43:43.636 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.636301 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:43:43.636 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.636317 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:43:43.637 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.636324 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:43:43.660 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.660994 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:43:43.661 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.661247 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:43:43.803 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:43.805171 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-04-09T11:43:44.074 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:44.076007 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:43:44.154 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_2 on smithi052 zapped 2022-04-09T11:43:44.172 INFO:teuthology.orchestra.run.smithi052.stderr:+ zap_completion=0 2022-04-09T11:43:44.172 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' 0 = 0 ']' 2022-04-09T11:43:44.173 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-04-09T11:43:44.606 INFO:teuthology.orchestra.run.smithi052.stderr:+ zap_completion= 2022-04-09T11:43:44.607 INFO:teuthology.orchestra.run.smithi052.stderr:+ sleep 3s 2022-04-09T11:43:45.575 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:45.576604 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:43:45.575 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:45.576668 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:43:45.576 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:45.576699 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-04-09T11:43:45.766 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:45.768574 I | op-mon: start running mons 2022-04-09T11:43:45.787 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:45.788977 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:43:45.798 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:45.797992 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:43:45.799 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:45.798017 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:43:45.799 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:45.798025 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:43:45.835 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:45.836459 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:43:46.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:46.025695 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:43:46.028 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:46.025998 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:43:46.028 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:46.029981 I | ceph-csi: successfully started CSI CephFS driver 2022-04-09T11:43:47.443 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:47.444608 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-09T11:43:47.448 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:47.450270 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-04-09T11:43:47.449 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:47.450291 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-09T11:43:47.457 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:47.455754 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-04-09T11:43:47.607 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' '' = 0 ']' 2022-04-09T11:43:47.608 INFO:teuthology.orchestra.run.smithi052.stderr:+ kubectl patch pv smithi052-lv-2 -p '{"spec":{"claimRef": null}}' 2022-04-09T11:43:47.807 INFO:teuthology.orchestra.run.smithi052.stdout:persistentvolume/smithi052-lv-2 patched 2022-04-09T11:43:47.811 INFO:teuthology.orchestra.run.smithi052.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-04-09T11:43:47.811 INFO:teuthology.orchestra.run.smithi052.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-04-09T11:43:48.116 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.117610 I | op-mon: targeting the mon count 1 2022-04-09T11:43:48.121 INFO:teuthology.orchestra.run.smithi052.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:48.122 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.123640 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:43:48.474 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.476373 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:43:48.475 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.476401 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:43:48.910 INFO:teuthology.orchestra.run.smithi052.stdout:['Success'] 2022-04-09T11:43:48.915 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.915323 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-09T11:43:48.916 INFO:tasks.rook.operator.smithi052.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", AllowUnsupported: true}, 2022-04-09T11:43:48.916 INFO:tasks.rook.operator.smithi052.stdout: Storage: v1.StorageScopeSpec{ 2022-04-09T11:43:48.916 INFO:tasks.rook.operator.smithi052.stdout: ... // 3 identical fields 2022-04-09T11:43:48.916 INFO:tasks.rook.operator.smithi052.stdout: Config: nil, 2022-04-09T11:43:48.916 INFO:tasks.rook.operator.smithi052.stdout: Selection: {}, 2022-04-09T11:43:48.917 INFO:tasks.rook.operator.smithi052.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-04-09T11:43:48.917 INFO:tasks.rook.operator.smithi052.stdout: {Name: "smithi052-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi052-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:48.917 INFO:tasks.rook.operator.smithi052.stdout: {Name: "smithi052-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi052-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:48.918 INFO:tasks.rook.operator.smithi052.stdout: {Name: "smithi052-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi052-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:48.918 INFO:tasks.rook.operator.smithi052.stdout:+ { 2022-04-09T11:43:48.918 INFO:tasks.rook.operator.smithi052.stdout:+ Name: "smithi052-lv-2", 2022-04-09T11:43:48.918 INFO:tasks.rook.operator.smithi052.stdout:+ Count: 1, 2022-04-09T11:43:48.919 INFO:tasks.rook.operator.smithi052.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-04-09T11:43:48.919 INFO:tasks.rook.operator.smithi052.stdout:+ }, 2022-04-09T11:43:48.919 INFO:tasks.rook.operator.smithi052.stdout: }, 2022-04-09T11:43:48.919 INFO:tasks.rook.operator.smithi052.stdout: }, 2022-04-09T11:43:48.919 INFO:tasks.rook.operator.smithi052.stdout: Annotations: nil, 2022-04-09T11:43:48.920 INFO:tasks.rook.operator.smithi052.stdout: Labels: nil, 2022-04-09T11:43:48.920 INFO:tasks.rook.operator.smithi052.stdout: ... // 20 identical fields 2022-04-09T11:43:48.920 INFO:tasks.rook.operator.smithi052.stdout: } 2022-04-09T11:43:48.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.915424 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-04-09T11:43:48.921 INFO:tasks.rook.operator.smithi052.stdout:I0409 11:43:48.915628 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-04-09T11:43:48.921 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.915833 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-04-09T11:43:48.924 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.923473 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-04-09T11:43:48.925 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.923518 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-04-09T11:43:48.929 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.929745 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-09T11:43:48.929 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.929769 I | operator: watching all namespaces for Ceph CRs 2022-04-09T11:43:48.929 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.929800 I | operator: setting up schemes 2022-04-09T11:43:48.932 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.932757 I | operator: setting up the controller-runtime manager 2022-04-09T11:43:48.942 INFO:teuthology.orchestra.run.smithi052.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-04-09T11:43:48.966 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.968522 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:43:48.967 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.968545 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:43:48.967 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.968605 W | op-mon: failed to set Rook and/or user-defined Ceph config options before starting mons; will retry after starting mons. failed to apply default Ceph configurations: failed to set one or more Ceph configs: failed to set ceph config in the centralized mon configuration database; you may need to use the rook-config-override ConfigMap. output: : context canceled 2022-04-09T11:43:48.968 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.968613 I | op-mon: checking for basic quorum with existing mons 2022-04-09T11:43:48.980 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.981824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-04-09T11:43:48.980 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:48.981895 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:43:49.017 INFO:teuthology.orchestra.run.smithi052.stdout:job.batch "rook-ceph-device-zap" deleted 2022-04-09T11:43:49.023 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.025166 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:43:49.026 INFO:teuthology.orchestra.run.smithi052.stdout:waiting for osd to come back up 2022-04-09T11:43:49.027 INFO:teuthology.orchestra.run.smithi052.stderr:+ num_osd=0 2022-04-09T11:43:49.027 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' 0 '!=' 4 ']' 2022-04-09T11:43:49.028 INFO:teuthology.orchestra.run.smithi052.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:43:49.028 INFO:teuthology.orchestra.run.smithi052.stderr:++ toolbox ceph osd stat 2022-04-09T11:43:49.028 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:43:49.028 INFO:teuthology.orchestra.run.smithi052.stderr:++ cut -f3 -d ' ' 2022-04-09T11:43:49.034 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.031046 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:43:49.036 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.031085 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:43:49.036 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.031097 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:43:49.048 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.049868 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-04-09 11:43:49.031156638 +0000 UTC m=+278.952740604 LastTransitionTime:2022-04-09 11:43:49.031156506 +0000 UTC m=+278.952740486}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:43:49.048 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.049886 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:43:49.134 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.135840 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-09T11:43:49.221 INFO:teuthology.orchestra.run.smithi052.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:49.240 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242033 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-09T11:43:49.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242071 I | ceph-cluster-controller: successfully started 2022-04-09T11:43:49.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242126 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-09T11:43:49.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242205 I | ceph-crashcollector-controller: successfully started 2022-04-09T11:43:49.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242279 I | ceph-block-pool-controller: successfully started 2022-04-09T11:43:49.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242341 I | ceph-object-store-user-controller: successfully started 2022-04-09T11:43:49.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242397 I | ceph-object-realm-controller: successfully started 2022-04-09T11:43:49.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242437 I | ceph-object-zonegroup-controller: successfully started 2022-04-09T11:43:49.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242545 I | ceph-object-zone-controller: successfully started 2022-04-09T11:43:49.247 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242805 I | ceph-object-controller: successfully started 2022-04-09T11:43:49.247 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.242954 I | ceph-file-controller: successfully started 2022-04-09T11:43:49.247 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243104 I | ceph-nfs-controller: successfully started 2022-04-09T11:43:49.247 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243260 I | ceph-rbd-mirror-controller: successfully started 2022-04-09T11:43:49.248 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243319 I | ceph-client-controller: successfully started 2022-04-09T11:43:49.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243419 I | ceph-filesystem-mirror-controller: successfully started 2022-04-09T11:43:49.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243516 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-09T11:43:49.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243610 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-09T11:43:49.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243695 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-09T11:43:49.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243781 I | ceph-bucket-topic: successfully started 2022-04-09T11:43:49.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243888 I | ceph-bucket-notification: successfully started 2022-04-09T11:43:49.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.243972 I | ceph-bucket-notification: successfully started 2022-04-09T11:43:49.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.244053 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-09T11:43:49.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.244148 I | blockpool-rados-namespace-controller: successfully started 2022-04-09T11:43:49.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.245145 I | operator: starting the controller-runtime manager 2022-04-09T11:43:49.347 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.348524 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-09T11:43:49.347 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.348570 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-09T11:43:49.347 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.348580 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-09T11:43:49.351 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.350624 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:43:49.357 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:49.359234 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-09T11:43:49.940 INFO:teuthology.orchestra.run.smithi052.stderr:+ num_osd=3 2022-04-09T11:43:49.940 INFO:teuthology.orchestra.run.smithi052.stderr:+ sleep 30s 2022-04-09T11:43:50.608 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.609846 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:43:50.609 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.609914 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-04-09T11:43:50.609 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.609927 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:43:50.609 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.609983 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:43:50.614 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.611559 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-04-09T11:43:50.614 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.611783 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-04-09T11:43:50.614 INFO:tasks.rook.operator.smithi052.stdout:I0409 11:43:50.611862 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:43:50.643 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.644302 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:43:50.751 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752516 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-04-09T11:43:50.755 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752550 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-04-09T11:43:50.756 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752558 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-04-09T11:43:50.756 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752564 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-04-09T11:43:50.756 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752570 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-04-09T11:43:50.756 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752577 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.6.0" (default) 2022-04-09T11:43:50.757 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752584 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-04-09T11:43:50.757 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752591 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-04-09T11:43:50.757 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752599 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-04-09T11:43:50.757 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752618 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-04-09T11:43:50.758 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752625 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-04-09T11:43:50.758 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752632 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-04-09T11:43:50.758 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752638 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-04-09T11:43:50.758 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752644 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-04-09T11:43:50.759 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752650 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-04-09T11:43:50.759 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752657 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.6.0" 2022-04-09T11:43:50.759 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752736 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:43:50.759 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.752755 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:43:50.787 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:50.789689 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-04-09T11:43:51.009 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:51.011472 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-04-09T11:43:54.087 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:54.088787 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-04-09T11:43:57.093 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:43:57.095016 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-04-09T11:44:00.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:00.101360 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-04-09T11:44:01.192 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.193980 I | ceph-csi: Detected ceph CSI image version: "v3.6.0" 2022-04-09T11:44:01.193 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194029 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-04-09T11:44:01.193 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194042 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-09T11:44:01.193 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194049 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-09T11:44:01.193 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194063 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-09T11:44:01.194 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194088 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-09T11:44:01.194 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194100 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-09T11:44:01.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194129 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-09T11:44:01.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194149 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-09T11:44:01.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194161 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-04-09T11:44:01.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194189 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-09T11:44:01.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194205 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-09T11:44:01.204 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194228 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:44:01.204 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194254 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:44:01.205 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194280 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-04-09T11:44:01.205 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194295 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-04-09T11:44:01.205 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194307 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-04-09T11:44:01.205 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194330 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-04-09T11:44:01.205 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194342 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-04-09T11:44:01.206 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194349 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:44:01.206 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194356 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:44:01.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194362 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-04-09T11:44:01.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194383 I | ceph-csi: Kubernetes version is 1.23 2022-04-09T11:44:01.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194392 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-04-09T11:44:01.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.194398 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-04-09T11:44:01.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.204865 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:44:01.208 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.204890 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:44:01.208 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.204894 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:44:01.208 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.204898 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:44:01.208 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.204901 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:44:01.209 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.204903 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:44:01.209 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.204906 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-04-09T11:44:01.911 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.913559 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:44:01.912 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.913584 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:44:01.912 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.913589 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:44:02.000 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:01.997403 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-04-09T11:44:02.165 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:02.167529 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:44:02.774 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:02.167554 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:44:02.775 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:02.167560 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-04-09T11:44:02.775 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:02.268022 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:44:02.776 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:02.268049 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:44:02.776 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:02.268058 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:44:02.776 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:02.340021 I | ceph-csi: successfully started CSI CephFS driver 2022-04-09T11:44:03.083 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:03.085211 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-09T11:44:03.087 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:03.089416 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-04-09T11:44:03.087 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:03.089437 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-09T11:44:03.093 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:03.093727 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-04-09T11:44:03.101 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:03.103799 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-04-09T11:44:10.619 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:10.620877 I | ceph-spec: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:44:10.619 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:10.620898 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:44:10.918 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:10.920616 I | ceph-spec: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:44:10.919 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:10.920647 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:44:11.138 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.139758 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:44:11.142 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.141110 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:44:11.283 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.285448 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:11.284 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.285573 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:11.284 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.285754 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:11.287 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.285917 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:11.691 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.693392 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:44:11.697 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.699651 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:44:11.747 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.749355 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-04-09 11:44:11.714050309 +0000 UTC m=+301.635634263 LastTransitionTime:2022-04-09 11:44:11.71405024 +0000 UTC m=+301.635634203}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:44:11.779 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.781488 I | op-mon: start running mons 2022-04-09T11:44:11.828 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.830766 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mons LastHeartbeatTime:2022-04-09 11:44:11.775693661 +0000 UTC m=+301.697277654 LastTransitionTime:2022-04-09 11:44:11.775693427 +0000 UTC m=+301.697277471}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:44:11.834 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:11.836939 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:44:12.100 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:12.102094 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:44:12.699 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:12.701559 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:12.707 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:12.703201 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:14.300 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:14.302177 I | op-mon: targeting the mon count 1 2022-04-09T11:44:14.306 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:14.308277 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:44:14.652 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:14.654282 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:44:14.652 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:14.654314 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:44:15.012 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:15.014617 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:44:15.013 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:15.014639 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:44:15.391 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:15.392956 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:44:15.391 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:15.392991 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:44:15.729 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:15.731869 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:44:15.730 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:15.731900 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:44:16.090 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.091842 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:44:16.090 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.091876 I | op-mon: checking for basic quorum with existing mons 2022-04-09T11:44:16.208 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.210455 I | op-mon: mon "a" ip is 10.97.79.62 2022-04-09T11:44:16.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.243808 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:44:16.313 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.315762 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:16.313 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.315937 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:16.716 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.718379 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-09T11:44:16.723 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.725617 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-09T11:44:16.723 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.725639 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:44:16.903 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:16.905864 I | op-mon: mons running: [a] 2022-04-09T11:44:17.327 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:17.329300 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:44:17.327 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:17.329334 I | op-mon: mons created: 1 2022-04-09T11:44:17.781 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:17.783446 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:44:17.788 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:17.791300 I | op-mon: mons running: [a] 2022-04-09T11:44:18.197 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:18.199521 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:44:18.198 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:18.199573 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:44:18.198 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:18.199674 I | op-mon: start running mons 2022-04-09T11:44:18.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:18.210256 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:44:18.499 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:18.501797 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:44:18.616 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:18.619103 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:44:19.044 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:19.046155 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:44:19.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:19.101344 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:19.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:19.101569 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:19.482 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:19.484029 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:44:19.940 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' 3 '!=' 4 ']' 2022-04-09T11:44:19.941 INFO:teuthology.orchestra.run.smithi052.stdout:waiting for osd to come back up 2022-04-09T11:44:19.942 INFO:teuthology.orchestra.run.smithi052.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:44:19.942 INFO:teuthology.orchestra.run.smithi052.stderr:++ toolbox ceph osd stat 2022-04-09T11:44:19.942 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:44:19.943 INFO:teuthology.orchestra.run.smithi052.stderr:++ cut -f3 -d ' ' 2022-04-09T11:44:20.008 INFO:teuthology.orchestra.run.smithi052.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:44:20.472 INFO:teuthology.orchestra.run.smithi052.stderr:+ num_osd=3 2022-04-09T11:44:20.472 INFO:teuthology.orchestra.run.smithi052.stderr:+ sleep 30s 2022-04-09T11:44:21.298 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:21.300099 I | op-mon: targeting the mon count 1 2022-04-09T11:44:21.304 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:21.306852 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:44:21.646 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:21.647976 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:44:21.646 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:21.648010 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:44:21.983 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:21.985407 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:44:21.983 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:21.985441 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:44:22.300 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:22.302278 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:44:22.300 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:22.302309 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:44:22.329 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:22.331377 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:44:22.329 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:22.331409 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:44:22.673 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:22.675158 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:44:22.673 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:22.675188 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:44:22.736 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:22.738319 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:44:22.736 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:22.738353 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:44:22.736 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:22.738361 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:44:23.028 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:23.030791 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:44:23.029 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:23.030824 I | op-mon: checking for basic quorum with existing mons 2022-04-09T11:44:23.196 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:23.198783 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:44:23.702 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:23.703277 I | op-mon: mon "a" ip is 10.97.79.62 2022-04-09T11:44:23.912 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:23.914629 I | op-mgr: start running mgr 2022-04-09T11:44:23.913 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:23.914664 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:44:24.498 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:24.500664 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:44:25.555 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:25.557891 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-09T11:44:25.562 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:25.565055 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-09T11:44:25.566 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:25.568589 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:44:25.699 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:25.701206 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:25.703 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:25.701583 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:26.510 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:26.512820 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-09T11:44:26.517 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:26.519958 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-09T11:44:26.517 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:26.519980 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:44:26.903 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:26.905616 I | op-mon: mons running: [a] 2022-04-09T11:44:27.299 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.301038 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-04-09T11:44:27.299 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.301115 I | op-mgr: successful modules: balancer 2022-04-09T11:44:27.316 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.318892 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:44:27.317 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.318917 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:44:27.325 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.328143 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:44:27.326 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.328163 I | op-mon: mons created: 1 2022-04-09T11:44:27.377 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.379543 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:44:27.498 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.500204 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-1" 2022-04-09T11:44:27.498 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.500238 I | op-osd: OSD PVC "smithi052-lv-1-data-0z6zxn" already exists 2022-04-09T11:44:27.498 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.500250 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-3" 2022-04-09T11:44:27.499 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.500263 I | op-osd: OSD PVC "smithi052-lv-3-data-0vswhb" already exists 2022-04-09T11:44:27.499 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.500319 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-4" 2022-04-09T11:44:27.499 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.500332 I | op-osd: OSD PVC "smithi052-lv-4-data-0fj58k" already exists 2022-04-09T11:44:27.499 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.500337 I | op-osd: creating 1 new PVCs for device set "smithi052-lv-2" 2022-04-09T11:44:27.698 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.700772 I | op-osd: successfully provisioned PVC "smithi052-lv-2-data-0tthhj" 2022-04-09T11:44:27.731 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.734140 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:44:27.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.734164 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-1-data-0z6zxn" because OSD daemon using the PVC already exists 2022-04-09T11:44:27.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.734171 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-3-data-0vswhb" 2022-04-09T11:44:27.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.734176 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-3-data-0vswhb" because OSD daemon using the PVC already exists 2022-04-09T11:44:27.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.734181 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-4-data-0fj58k" 2022-04-09T11:44:27.733 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.734186 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-4-data-0fj58k" because OSD daemon using the PVC already exists 2022-04-09T11:44:27.733 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.734191 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-2-data-0tthhj" 2022-04-09T11:44:27.931 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:27.933492 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:44:28.143 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.145082 I | op-osd: started OSD provisioning job for PVC "smithi052-lv-2-data-0tthhj" 2022-04-09T11:44:28.143 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.145106 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:44:28.143 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.145110 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:44:28.303 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.305992 I | op-mon: mons running: [a] 2022-04-09T11:44:28.503 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.505790 I | op-osd: OSD orchestration status for PVC smithi052-lv-2-data-0tthhj is "starting" 2022-04-09T11:44:28.604 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.607163 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:44:28.792 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.786346 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:44:28.793 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.786388 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:44:28.874 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.876607 I | op-osd: updating OSD 1 on PVC "smithi052-lv-3-data-0vswhb" 2022-04-09T11:44:28.875 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:28.876631 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-3-data-0vswhb" 2022-04-09T11:44:29.377 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.380118 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:44:29.381 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.384300 I | op-osd: updating OSD 2 on PVC "smithi052-lv-4-data-0fj58k" 2022-04-09T11:44:29.381 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.384317 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-4-data-0fj58k" 2022-04-09T11:44:29.570 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.572468 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:44:29.575 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.576253 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:44:29.578 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.578375 I | op-mgr: successful modules: prometheus 2022-04-09T11:44:29.578 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.580695 I | op-mgr: successful modules: dashboard 2022-04-09T11:44:29.615 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.617749 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:44:29.741 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.743725 I | op-osd: updating OSD 3 on PVC "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:44:29.741 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:29.743757 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:44:30.007 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:30.009052 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:44:30.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:30.453311 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:44:31.125 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:31.127554 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:44:31.148 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:31.151249 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:44:31.149 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:31.151271 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:44:31.703 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:31.705897 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:44:31.704 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:31.705926 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:44:31.704 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:31.705933 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:44:32.145 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:32.147459 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:44:32.973 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:32.975450 I | op-mgr: start running mgr 2022-04-09T11:44:32.973 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:32.975480 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:44:33.913 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:33.915782 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-09T11:44:33.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:33.922847 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-09T11:44:33.924 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:33.926104 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:44:34.290 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.292788 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-04-09T11:44:34.291 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.292850 I | op-mgr: successful modules: balancer 2022-04-09T11:44:34.308 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.306435 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:44:34.308 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.306471 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:44:34.392 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.394745 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:44:34.408 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.399660 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-1" 2022-04-09T11:44:34.408 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.399696 I | op-osd: OSD PVC "smithi052-lv-1-data-0z6zxn" already exists 2022-04-09T11:44:34.409 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.399709 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-3" 2022-04-09T11:44:34.409 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.399720 I | op-osd: OSD PVC "smithi052-lv-3-data-0vswhb" already exists 2022-04-09T11:44:34.409 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.399729 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-4" 2022-04-09T11:44:34.409 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.399738 I | op-osd: OSD PVC "smithi052-lv-4-data-0fj58k" already exists 2022-04-09T11:44:34.410 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.399746 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-2" 2022-04-09T11:44:34.410 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.399754 I | op-osd: OSD PVC "smithi052-lv-2-data-0tthhj" already exists 2022-04-09T11:44:34.410 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.408595 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:44:34.410 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.408621 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-1-data-0z6zxn" because OSD daemon using the PVC already exists 2022-04-09T11:44:34.410 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.408628 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-3-data-0vswhb" 2022-04-09T11:44:34.411 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.408633 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-3-data-0vswhb" because OSD daemon using the PVC already exists 2022-04-09T11:44:34.411 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.408637 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-4-data-0fj58k" 2022-04-09T11:44:34.411 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.408642 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-4-data-0fj58k" because OSD daemon using the PVC already exists 2022-04-09T11:44:34.411 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.408647 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-2-data-0tthhj" 2022-04-09T11:44:34.418 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.419686 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi052-lv-2-data-0tthhj. Status={Conditions:[] StartTime:2022-04-09 11:44:28 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-04-09T11:44:34.418 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.419844 I | op-osd: started OSD provisioning job for PVC "smithi052-lv-2-data-0tthhj" 2022-04-09T11:44:34.419 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.419887 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:44:34.419 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.419913 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:44:34.421 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.422221 I | op-osd: OSD orchestration status for PVC smithi052-lv-2-data-0tthhj is "starting" 2022-04-09T11:44:34.521 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.523687 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:44:34.530 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.532205 I | op-osd: updating OSD 1 on PVC "smithi052-lv-3-data-0vswhb" 2022-04-09T11:44:34.530 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.532230 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-3-data-0vswhb" 2022-04-09T11:44:34.729 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.731448 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:44:34.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.735378 I | op-osd: updating OSD 2 on PVC "smithi052-lv-4-data-0fj58k" 2022-04-09T11:44:34.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:34.735396 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-4-data-0fj58k" 2022-04-09T11:44:35.210 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:35.213033 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:44:35.215 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:35.217932 I | op-osd: updating OSD 3 on PVC "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:44:35.215 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:35.217958 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:44:36.191 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:36.193318 I | op-mgr: successful modules: dashboard 2022-04-09T11:44:36.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:36.197611 I | op-mgr: successful modules: prometheus 2022-04-09T11:44:36.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:36.197775 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:44:36.472 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:36.474976 I | op-osd: OSD orchestration status for node smithi052-lv-2-data-0tthhj is "orchestrating" 2022-04-09T11:44:36.473 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:36.475010 I | op-osd: OSD orchestration status for node smithi052-lv-2-data-0tthhj is "orchestrating" 2022-04-09T11:44:36.866 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:36.868701 I | op-osd: OSD orchestration status for PVC smithi052-lv-2-data-0tthhj is "orchestrating" 2022-04-09T11:44:36.866 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:36.868729 I | op-osd: OSD orchestration status for PVC smithi052-lv-2-data-0tthhj is "orchestrating" 2022-04-09T11:44:37.488 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:37.490984 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:44:42.979 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:42.980875 I | op-osd: OSD orchestration status for PVC smithi052-lv-2-data-0tthhj is "completed" 2022-04-09T11:44:42.979 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:42.980906 I | op-osd: creating OSD 0 on PVC "smithi052-lv-2-data-0tthhj" 2022-04-09T11:44:42.979 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:42.980930 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-2-data-0tthhj" 2022-04-09T11:44:42.980 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:42.980975 I | op-osd: OSD orchestration status for PVC smithi052-lv-2-data-0tthhj is "completed" 2022-04-09T11:44:42.980 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:42.981001 I | op-osd: creating OSD 0 on PVC "smithi052-lv-2-data-0tthhj" 2022-04-09T11:44:42.980 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:42.981017 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-2-data-0tthhj" 2022-04-09T11:44:43.033 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:43.035361 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 0 on PVC "smithi052-lv-2-data-0tthhj" LastHeartbeatTime:2022-04-09 11:44:43.002184861 +0000 UTC m=+332.923768843 LastTransitionTime:2022-04-09 11:44:43.002184713 +0000 UTC m=+332.923768709}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:44:43.254 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:43.256812 E | op-osd: failed to create OSD 0 on PVC "smithi052-lv-2-data-0tthhj": failed to create deployment for OSD 0 on PVC "smithi052-lv-2-data-0tthhj": deployments.apps "rook-ceph-osd-0" already exists 2022-04-09T11:44:43.989 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:43.991612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-04-09T11:44:43.989 INFO:tasks.rook.operator.smithi052.stdout:failed to create OSD 0 on PVC "smithi052-lv-2-data-0tthhj": failed to create deployment for OSD 0 on PVC "smithi052-lv-2-data-0tthhj": deployments.apps "rook-ceph-osd-0" already exists 2022-04-09T11:44:43.994 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:43.997086 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:44:44.139 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:44.142033 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-09T11:44:44.140 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:44.142054 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:44:44.181 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:44.183858 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:44:44.256 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:44.258817 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:44:50.473 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' 3 '!=' 4 ']' 2022-04-09T11:44:50.473 INFO:teuthology.orchestra.run.smithi052.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:44:50.474 INFO:teuthology.orchestra.run.smithi052.stdout:waiting for osd to come back up 2022-04-09T11:44:50.475 INFO:teuthology.orchestra.run.smithi052.stderr:++ toolbox ceph osd stat 2022-04-09T11:44:50.475 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:44:50.475 INFO:teuthology.orchestra.run.smithi052.stderr:++ cut -f3 -d ' ' 2022-04-09T11:44:50.564 INFO:teuthology.orchestra.run.smithi052.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:44:51.624 INFO:teuthology.orchestra.run.smithi052.stderr:+ num_osd=3 2022-04-09T11:44:51.624 INFO:teuthology.orchestra.run.smithi052.stderr:+ sleep 30s 2022-04-09T11:44:56.878 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:56.874899 I | ceph-spec: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:44:56.878 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:56.874992 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:44:57.219 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:57.221710 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:44:57.448 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:57.449083 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:57.448 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:57.449490 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:57.865 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:57.868242 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:44:58.469 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:58.471987 I | op-mon: start running mons 2022-04-09T11:44:58.525 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:58.527470 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:44:58.999 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:59.001358 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:44:59.006 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:59.008912 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:59.006 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:44:59.009064 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:45:00.237 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:00.240425 I | op-mon: targeting the mon count 1 2022-04-09T11:45:00.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:00.244538 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:45:00.606 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:00.608881 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:45:00.607 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:00.608911 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:45:01.056 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:01.059345 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:45:01.057 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:01.059379 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:45:01.392 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:01.394678 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:45:01.392 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:01.394712 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:45:01.751 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:01.754024 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:45:01.751 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:01.754045 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:45:02.146 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.148608 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:45:02.146 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.148627 I | op-mon: checking for basic quorum with existing mons 2022-04-09T11:45:02.829 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.832235 I | op-mon: mon "a" ip is 10.97.79.62 2022-04-09T11:45:02.960 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.963404 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.79.62:6789"]}] data:a=10.97.79.62:6789 mapping:{"node":{"a":{"Name":"smithi052","Hostname":"smithi052","Address":"172.21.15.52"}}} maxMonId:0] 2022-04-09T11:45:02.968 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.971135 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:45:02.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.971329 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:45:02.981 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.984777 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-09T11:45:02.989 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.992800 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-09T11:45:02.990 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.992824 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:45:02.996 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:02.999064 I | op-mon: mons running: [a] 2022-04-09T11:45:03.440 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:03.443159 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:45:03.440 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:03.443188 I | op-mon: mons created: 1 2022-04-09T11:45:03.855 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:03.857891 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:45:03.864 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:03.866884 I | op-mon: mons running: [a] 2022-04-09T11:45:04.283 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:04.286107 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:45:04.283 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:04.286142 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:45:04.708 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:04.710908 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:45:05.155 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:05.157620 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:45:05.575 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:05.578739 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:45:06.036 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:06.038665 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:45:06.036 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:06.038691 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:45:06.476 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:06.479000 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:45:06.476 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:06.479024 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:45:06.477 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:06.479029 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:45:06.915 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:06.918170 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:45:06.970 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:06.972634 I | op-mgr: start running mgr 2022-04-09T11:45:06.970 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:06.972672 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:45:07.422 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:07.425162 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-09T11:45:07.435 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:07.435322 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-09T11:45:07.436 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:07.438980 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:45:08.022 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.024547 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-04-09T11:45:08.022 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.024669 I | op-mgr: successful modules: balancer 2022-04-09T11:45:08.036 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.039009 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:45:08.039 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.039046 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:45:08.070 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.073471 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:45:08.220 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.223246 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-1" 2022-04-09T11:45:08.224 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.227330 I | op-osd: OSD PVC "smithi052-lv-1-data-0z6zxn" already exists 2022-04-09T11:45:08.228 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.231049 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-3" 2022-04-09T11:45:08.231 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.234809 I | op-osd: OSD PVC "smithi052-lv-3-data-0vswhb" already exists 2022-04-09T11:45:08.236 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.238727 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-4" 2022-04-09T11:45:08.239 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.242388 I | op-osd: OSD PVC "smithi052-lv-4-data-0fj58k" already exists 2022-04-09T11:45:08.240 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.242428 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi052-lv-2" 2022-04-09T11:45:08.240 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.242441 I | op-osd: OSD PVC "smithi052-lv-2-data-0tthhj" already exists 2022-04-09T11:45:08.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252177 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:45:08.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252197 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-1-data-0z6zxn" because OSD daemon using the PVC already exists 2022-04-09T11:45:08.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252202 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-3-data-0vswhb" 2022-04-09T11:45:08.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252205 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-3-data-0vswhb" because OSD daemon using the PVC already exists 2022-04-09T11:45:08.250 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252208 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-4-data-0fj58k" 2022-04-09T11:45:08.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252211 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-4-data-0fj58k" because OSD daemon using the PVC already exists 2022-04-09T11:45:08.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252214 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-2-data-0tthhj" 2022-04-09T11:45:08.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252218 I | op-osd: skipping OSD prepare job creation for PVC "smithi052-lv-2-data-0tthhj" because OSD daemon using the PVC already exists 2022-04-09T11:45:08.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252220 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:45:08.252 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.252223 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:45:08.522 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.524987 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:45:08.528 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.531022 I | op-osd: updating OSD 0 on PVC "smithi052-lv-2-data-0tthhj" 2022-04-09T11:45:08.528 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.531051 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-2-data-0tthhj" 2022-04-09T11:45:08.706 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.708581 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:45:08.712 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.715508 I | op-osd: updating OSD 1 on PVC "smithi052-lv-3-data-0vswhb" 2022-04-09T11:45:08.713 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.715547 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-3-data-0vswhb" 2022-04-09T11:45:08.869 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.871888 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:45:08.875 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.878030 I | op-osd: updating OSD 2 on PVC "smithi052-lv-4-data-0fj58k" 2022-04-09T11:45:08.875 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:08.878057 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-4-data-0fj58k" 2022-04-09T11:45:09.060 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:09.062722 I | op-osd: skipping osd checks for ok-to-stop 2022-04-09T11:45:09.130 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:09.133160 I | op-osd: updating OSD 3 on PVC "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:45:09.131 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:09.133195 I | op-osd: OSD will have its main bluestore block on "smithi052-lv-1-data-0z6zxn" 2022-04-09T11:45:09.206 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:09.209332 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:45:09.213 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:09.216694 I | op-mgr: successful modules: dashboard 2022-04-09T11:45:09.216 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:09.216962 I | op-mgr: successful modules: prometheus 2022-04-09T11:45:10.230 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:10.233503 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-09T11:45:10.231 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:10.233530 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:45:10.616 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:10.619014 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:45:21.625 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' 3 '!=' 4 ']' 2022-04-09T11:45:21.626 INFO:teuthology.orchestra.run.smithi052.stdout:waiting for osd to come back up 2022-04-09T11:45:21.627 INFO:teuthology.orchestra.run.smithi052.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:45:21.627 INFO:teuthology.orchestra.run.smithi052.stderr:++ toolbox ceph osd stat 2022-04-09T11:45:21.627 INFO:teuthology.orchestra.run.smithi052.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:45:21.627 INFO:teuthology.orchestra.run.smithi052.stderr:++ cut -f3 -d ' ' 2022-04-09T11:45:21.742 INFO:teuthology.orchestra.run.smithi052.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:45:22.219 INFO:teuthology.orchestra.run.smithi052.stderr:+ num_osd=4 2022-04-09T11:45:22.219 INFO:teuthology.orchestra.run.smithi052.stderr:+ sleep 30s 2022-04-09T11:45:52.223 INFO:teuthology.orchestra.run.smithi052.stderr:+ '[' 4 '!=' 4 ']' 2022-04-09T11:45:52.225 INFO:teuthology.run_tasks:Running task rook.shell... 2022-04-09T11:45:52.237 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch status 2022-04-09T11:45:52.916 INFO:teuthology.orchestra.run.smithi052.stdout:Backend: rook 2022-04-09T11:45:52.916 INFO:teuthology.orchestra.run.smithi052.stdout:Available: Yes 2022-04-09T11:45:52.947 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ps 2022-04-09T11:45:53.501 INFO:teuthology.orchestra.run.smithi052.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-04-09T11:45:53.501 INFO:teuthology.orchestra.run.smithi052.stdout:crashcollector.crash smithi052 running (3m) 0s ago 3m - - 2abf022dd4de 2022-04-09T11:45:53.501 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.a smithi052 running (4m) 0s ago 4m - - 2abf022dd4de 2022-04-09T11:45:53.502 INFO:teuthology.orchestra.run.smithi052.stdout:mon.a smithi052 running (5m) 0s ago 5m - - 2abf022dd4de 2022-04-09T11:45:53.502 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 smithi052 running (70s) 0s ago 70s - - 2abf022dd4de 2022-04-09T11:45:53.502 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 smithi052 running (3m) 0s ago 3m - - 2abf022dd4de 2022-04-09T11:45:53.502 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 smithi052 running (3m) 0s ago 3m - - 2abf022dd4de 2022-04-09T11:45:53.503 INFO:teuthology.orchestra.run.smithi052.stdout:osd.3 smithi052 running (3m) 0s ago 3m - - 2abf022dd4de 2022-04-09T11:45:53.521 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls 2022-04-09T11:45:54.157 INFO:teuthology.orchestra.run.smithi052.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-04-09T11:45:54.158 INFO:teuthology.orchestra.run.smithi052.stdout:crash 1/1 0s ago 3m * 2022-04-09T11:45:54.158 INFO:teuthology.orchestra.run.smithi052.stdout:mgr 1/1 0s ago 4m count:1 2022-04-09T11:45:54.158 INFO:teuthology.orchestra.run.smithi052.stdout:mon 1/1 0s ago 5m count:1 2022-04-09T11:45:54.160 INFO:teuthology.orchestra.run.smithi052.stdout:osd 8 0s ago 3m 2022-04-09T11:45:54.160 INFO:teuthology.orchestra.run.smithi052.stdout:osd.all-available-devices 0 0s ago - * 2022-04-09T11:45:54.180 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch host ls 2022-04-09T11:45:54.686 INFO:teuthology.orchestra.run.smithi052.stdout:HOST ADDR LABELS STATUS 2022-04-09T11:45:54.751 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 172.21.15.52/smithi052 2022-04-09T11:45:54.752 INFO:teuthology.orchestra.run.smithi052.stdout:1 hosts in cluster 2022-04-09T11:45:54.753 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch device ls 2022-04-09T11:45:55.286 INFO:teuthology.orchestra.run.smithi052.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-04-09T11:45:55.287 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-09T11:45:55.287 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-09T11:45:55.287 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-09T11:45:55.288 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-09T11:45:55.313 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch apply rgw foo 2022-04-09T11:45:55.892 INFO:teuthology.orchestra.run.smithi052.stdout:Created 2022-04-09T11:45:55.894 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:55.894594 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-04-09T11:45:55.915 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch apply mds foo 2022-04-09T11:45:55.950 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:55.950839 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:45:55.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:55.950899 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:45:56.687 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:56.687602 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-04-09T11:45:56.688 INFO:teuthology.orchestra.run.smithi052.stdout:Created 2022-04-09T11:45:56.710 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch apply rbd-mirror 2022-04-09T11:45:56.994 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:56.994520 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:45:56.995 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:56.994568 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:45:57.946 INFO:teuthology.orchestra.run.smithi052.stdout:Success 2022-04-09T11:45:57.968 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch apply nfs foo --port 12777 2022-04-09T11:45:58.151 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:58.151368 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:45:58.152 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:45:58.151464 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:46:02.692 INFO:teuthology.orchestra.run.smithi052.stdout:Unable to create NFS daemon, check logs for more traceback 2022-04-09T11:46:02.692 INFO:teuthology.orchestra.run.smithi052.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-04-09T11:46:02.715 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-04-09T11:46:02.726 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:46:03.880 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:46:03.880 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:03.795668Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:03.795668Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:46:03.795668Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:46:03.795668Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:03.795668Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:46:03.795668Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-04-09T11:46:03.795668Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:03.795668Z", "running": 0, "size": 1}}] 2022-04-09T11:46:03.900 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch rm mds.foo 2022-04-09T11:46:05.909 INFO:teuthology.orchestra.run.smithi052.stdout:Removed cephfilesystems/foo 2022-04-09T11:46:05.928 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch rm rbd-mirror 2022-04-09T11:46:06.994 INFO:teuthology.orchestra.run.smithi052.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-04-09T11:46:07.016 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch rm rgw.foo 2022-04-09T11:46:08.882 INFO:teuthology.orchestra.run.smithi052.stdout:Removed cephobjectstores/foo 2022-04-09T11:46:08.903 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:46:10.053 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:10.052913 I | ceph-spec: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:46:11.085 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:11.084197 I | ceph-spec: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:46:11.105 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:46:11.105 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:09.856651Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:09.856651Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:46:09.856651Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:46:09.856651Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:09.856651Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:46:09.856651Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:09.856651Z", "running": 0, "size": 1}}] 2022-04-09T11:46:11.882 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:11.881559 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:46:12.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:12.361187 I | ceph-object-controller: ceph object store gateway service running at 10.108.1.124 2022-04-09T11:46:12.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:12.361216 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:46:12.715 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:12.713743 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-04-09T11:46:13.003 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:13.001918 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:46:13.642 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:13.610274 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-04-09T11:46:14.392 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:14.390849 I | op-mds: setting mds config flags 2022-04-09T11:46:14.392 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:14.390875 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-04-09T11:46:15.045 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:15.044346 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-04-09T11:46:15.058 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:15.055948 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-04-09T11:46:15.375 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:15.367795 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi052": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:46:15.932 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:15.929624 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi052": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:46:16.344 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:16.342853 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi052": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:46:16.644 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:16.643593 I | op-mds: setting mds config flags 2022-04-09T11:46:16.645 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:16.643619 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-04-09T11:46:19.873 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:19.871485 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-04-09T11:46:21.127 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:46:22.545 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:46:22.546 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:22.453512Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:22.453512Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:46:22.453512Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:46:22.453512Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:22.453512Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:46:22.453512Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:22.453512Z", "running": 0, "size": 1}}] 2022-04-09T11:46:22.837 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:22.835511 I | ceph-file-controller: creating filesystem "foo" 2022-04-09T11:46:23.179 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:23.178177 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-04-09T11:46:23.184 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:23.180418 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-04-09T11:46:23.187 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:23.183861 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-04-09T11:46:23.187 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:23.184142 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-04-09T11:46:23.192 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:23.187600 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-04-09T11:46:24.864 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:24.858967 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-04-09T11:46:25.191 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:25.189596 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:27.488 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:27.485964 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-04-09T11:46:30.339 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:30.337097 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:30.327+0000 7f5f514e1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:30.339 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:30.327+0000 7f5f514e1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:30.339 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:30.327+0000 7f5f514e1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:30.340 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:30.373 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:30.372075 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:31.128 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:31.126416 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:31.115+0000 7f162a5d6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:31.128 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:31.115+0000 7f162a5d6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:31.129 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:31.115+0000 7f162a5d6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:31.129 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:31.246 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:31.244291 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:32.273 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:32.271468 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:32.259+0000 7f44b8365380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:32.274 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:32.259+0000 7f44b8365380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:32.274 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:32.259+0000 7f44b8365380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:32.274 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:32.287 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:32.285874 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:32.562 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:46:33.285 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:33.171230 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:33.139+0000 7fafe8ee4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:33.286 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:33.139+0000 7fafe8ee4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:33.286 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:33.139+0000 7fafe8ee4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:33.286 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:33.524 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:33.522643 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:33.838 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:46:33.838 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:33.211526Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:33.211526Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:46:33.211526Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:46:33.211526Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:33.211526Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:46:33.211526Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:33.211526Z", "running": 0, "size": 1}}] 2022-04-09T11:46:33.854 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:33.852276 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-04-09T11:46:34.193 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:34.191651 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:34.183+0000 7f964f12d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:34.194 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:34.183+0000 7f964f12d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:34.194 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:34.183+0000 7f964f12d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:34.194 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:34.199 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:34.195515 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:34.577 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:34.575132 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-04-09T11:46:34.924 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:34.921346 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:34.907+0000 7fa1ae852380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:34.925 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:34.907+0000 7fa1ae852380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:34.925 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:34.907+0000 7fa1ae852380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:34.925 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:34.979 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:34.977455 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:35.576 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:35.573879 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:35.563+0000 7f33cfc5d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:35.576 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:35.563+0000 7f33cfc5d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:35.576 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:35.563+0000 7f33cfc5d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:35.577 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:35.604 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:35.602067 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:36.362 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:36.359831 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:36.351+0000 7fe68faa2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:36.362 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:36.351+0000 7fe68faa2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:36.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:36.351+0000 7fe68faa2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:36.363 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:36.370 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:36.368887 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:36.960 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:36.952541 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-04-09T11:46:36.960 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:36.952569 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-04-09T11:46:37.087 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:37.084490 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:37.075+0000 7f14bbb3f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:37.087 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:37.075+0000 7f14bbb3f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:37.087 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:37.075+0000 7f14bbb3f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:37.087 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:37.649 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:37.647166 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:38.121 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:38.118687 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:38.125 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:38.122046 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-04-09T11:46:38.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:38.239628 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:38.231+0000 7f31491bd380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:38.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:38.231+0000 7f31491bd380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:38.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:38.231+0000 7f31491bd380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:38.242 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:38.907 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:38.904789 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-04-09T11:46:39.329 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:39.327079 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-04-09T11:46:39.329 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:39.327107 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-04-09T11:46:39.330 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:39.327114 I | cephclient: deleting ceph auth "mds.foo-a" 2022-04-09T11:46:39.850 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:39.848622 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-04-09T11:46:40.325 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:40.323267 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-04-09T11:46:40.837 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:40.834617 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-04-09T11:46:40.852 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:40.834647 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-04-09T11:46:40.852 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:40.834654 I | cephclient: deleting ceph auth "mds.foo-b" 2022-04-09T11:46:41.421 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:41.419328 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-04-09T11:46:41.422 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:41.419362 I | ceph-file-controller: downing filesystem "foo" 2022-04-09T11:46:42.011 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:42.009583 I | ceph-file-controller: downed filesystem "foo" 2022-04-09T11:46:43.648 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:43.646421 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:43.859 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:46:45.487 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:46:45.487 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:15.000000Z", "last_refresh": "2022-04-09T11:46:44.437845Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:44.437845Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:46:44.437845Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:46:44.437845Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:44.437845Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:46:44.437845Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:44.437845Z", "running": 0, "size": 1}}] 2022-04-09T11:46:46.068 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:46.066308 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:46.058+0000 7fa36962f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:46.069 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:46.058+0000 7fa36962f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:46.069 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:46.058+0000 7fa36962f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:46.069 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:46.734 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:46.732215 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-04-09T11:46:46.734 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:46.732233 I | cephclient: purging pool "foo-metadata" (id=8) 2022-04-09T11:46:49.504 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:49.502491 I | cephclient: purge completed for pool "foo-metadata" 2022-04-09T11:46:49.941 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:49.938485 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-04-09T11:46:49.941 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:49.938522 I | cephclient: purging pool "foo-data0" (id=10) 2022-04-09T11:46:52.837 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:52.832758 I | cephclient: purge completed for pool "foo-data0" 2022-04-09T11:46:52.838 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:52.832845 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-04-09T11:46:52.884 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:52.880591 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-04-09T11:46:52.885 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:52.880994 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-04-09T11:46:52.936 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:52.933919 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-04-09T11:46:52.943 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:52.938913 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-04-09T11:46:55.508 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:46:56.365 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:56.362147 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:46:56.926 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:46:56.927 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:15.000000Z", "last_refresh": "2022-04-09T11:46:56.796061Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:46:56.796061Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:46:56.796061Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:46:56.796061Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:46:56.796061Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:46:56.796061Z", "running": 0, "size": 1}}] 2022-04-09T11:46:57.261 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:57.258315 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:46:57.250+0000 7f666ebe1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:46:57.261 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:57.250+0000 7f666ebe1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:46:57.261 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:46:57.250+0000 7f666ebe1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:46:57.262 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:46:57.642 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:46:57.639902 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi052": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:47:06.957 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:47:07.861 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:47:07.861 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:15.000000Z", "last_refresh": "2022-04-09T11:47:07.767299Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:47:07.767299Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:47:07.767299Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:47:07.767299Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:47:07.767299Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:47:07.767299Z", "running": 0, "size": 1}}] 2022-04-09T11:47:17.751 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:47:17.745849 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:47:17.881 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:47:18.420 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:47:18.417506 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:47:18.410+0000 7f5b6bd21380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:47:18.420 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:47:18.410+0000 7f5b6bd21380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:47:18.420 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:47:18.410+0000 7f5b6bd21380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:47:18.421 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:47:18.529 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:47:18.530 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:47:18.431739Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:47:18.431739Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:47:18.431739Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:47:18.431739Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:47:18.431739Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:47:18.431739Z", "running": 0, "size": 1}}] 2022-04-09T11:47:28.551 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:47:29.188 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:47:29.188 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:47:29.084665Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:47:29.084665Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:47:29.084665Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:47:29.084665Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:47:29.084665Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:47:29.084665Z", "running": 0, "size": 1}}] 2022-04-09T11:47:39.215 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:47:39.898 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:47:39.899 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:47:39.756020Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:47:39.756020Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:47:39.756020Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:47:39.756020Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:47:39.756020Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:47:39.756020Z", "running": 0, "size": 1}}] 2022-04-09T11:47:49.925 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:47:50.580 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:47:50.580 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:47:50.481259Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:47:50.481259Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:47:50.481259Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:47:50.481259Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:47:50.481259Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:47:50.481259Z", "running": 0, "size": 1}}] 2022-04-09T11:47:59.388 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:47:59.385001 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:48:00.054 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:48:00.051236 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:48:00.041+0000 7f0ce5859380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:48:00.055 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:48:00.041+0000 7f0ce5859380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:48:00.055 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:48:00.041+0000 7f0ce5859380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:48:00.055 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:48:00.605 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:48:01.279 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:48:01.279 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:48:01.165917Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:48:01.165917Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:48:01.165917Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:48:01.165917Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:48:01.165917Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:48:01.165917Z", "running": 0, "size": 1}}] 2022-04-09T11:48:11.299 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:48:12.012 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:48:12.013 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:48:11.903815Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:48:11.903815Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:48:11.903815Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:48:11.903815Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:48:11.903815Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:48:11.903815Z", "running": 0, "size": 1}}] 2022-04-09T11:48:22.056 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:48:22.709 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:48:22.709 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:48:22.550328Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:48:22.550328Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:48:22.550328Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:48:22.550328Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:48:22.550328Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:48:22.550328Z", "running": 0, "size": 1}}] 2022-04-09T11:48:32.732 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:48:33.347 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:48:33.347 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:48:33.236259Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:48:33.236259Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:48:33.236259Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:48:33.236259Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:48:33.236259Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:48:33.236259Z", "running": 0, "size": 1}}] 2022-04-09T11:48:43.369 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:48:44.001 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:48:44.002 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:48:43.901410Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:48:43.901410Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:48:43.901410Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:48:43.901410Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:48:43.901410Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:48:43.901410Z", "running": 0, "size": 1}}] 2022-04-09T11:48:54.026 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:48:54.656 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:48:54.657 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:48:54.550073Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:48:54.550073Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:48:54.550073Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:48:54.550073Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:48:54.550073Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:48:54.550073Z", "running": 0, "size": 1}}] 2022-04-09T11:49:04.681 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:49:05.335 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:49:05.336 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:49:05.235350Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:49:05.235350Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:49:05.235350Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:49:05.235350Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:49:05.235350Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:49:05.235350Z", "running": 0, "size": 1}}] 2022-04-09T11:49:15.361 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:49:15.968 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:49:15.969 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:49:15.858038Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:49:15.858038Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:49:15.858038Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:49:15.858038Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:49:15.858038Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:49:15.858038Z", "running": 0, "size": 1}}] 2022-04-09T11:49:21.995 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:49:21.992688 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:49:22.621 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:49:22.618053 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:49:22.608+0000 7fe567b6d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:49:22.622 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:49:22.608+0000 7fe567b6d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:49:22.622 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:49:22.608+0000 7fe567b6d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:49:22.622 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:49:25.993 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:49:29.431 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:49:29.431 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:49:29.321003Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:49:29.321003Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:49:29.321003Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:49:29.321003Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:49:29.321003Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:49:29.321003Z", "running": 0, "size": 1}}] 2022-04-09T11:49:39.455 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:49:40.133 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:49:40.133 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:49:40.023576Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:49:40.023576Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:49:40.023576Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:49:40.023576Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:49:40.023576Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:49:40.023576Z", "running": 0, "size": 1}}] 2022-04-09T11:49:50.157 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:49:50.817 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:49:50.817 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:49:50.712318Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:49:50.712318Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:49:50.712318Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:49:50.712318Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:49:50.712318Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:49:50.712318Z", "running": 0, "size": 1}}] 2022-04-09T11:50:00.844 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:50:01.863 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:50:01.863 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:50:01.757846Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:50:01.757846Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:50:01.757846Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:50:01.757846Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:50:01.757846Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:50:01.757846Z", "running": 0, "size": 1}}] 2022-04-09T11:50:11.888 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:50:12.509 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:50:12.510 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:50:12.401196Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:50:12.401196Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:50:12.401196Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:50:12.401196Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:50:12.401196Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:50:12.401196Z", "running": 0, "size": 1}}] 2022-04-09T11:50:22.533 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:50:23.170 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:50:23.171 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:50:23.063141Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:50:23.063141Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:50:23.063141Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:50:23.063141Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:50:23.063141Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:50:23.063141Z", "running": 0, "size": 1}}] 2022-04-09T11:50:33.230 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:50:33.851 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:50:33.852 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:50:33.752512Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:50:33.752512Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:50:33.752512Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:50:33.752512Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:50:33.752512Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:50:33.752512Z", "running": 0, "size": 1}}] 2022-04-09T11:50:43.876 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:50:44.530 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:50:44.531 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:50:44.410133Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:50:44.410133Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:50:44.410133Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:50:44.410133Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:50:44.410133Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:50:44.410133Z", "running": 0, "size": 1}}] 2022-04-09T11:50:54.552 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:50:55.224 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:50:55.225 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:50:55.119594Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:50:55.119594Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:50:55.119594Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:50:55.119594Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:50:55.119594Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:50:55.119594Z", "running": 0, "size": 1}}] 2022-04-09T11:51:05.250 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:51:06.183 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:51:06.184 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:51:05.795767Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:51:05.795767Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:51:05.795767Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:51:05.795767Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:51:05.795767Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:51:05.795767Z", "running": 0, "size": 1}}] 2022-04-09T11:51:16.207 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:51:16.902 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:51:16.903 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:51:16.730186Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:51:16.730186Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:51:16.730186Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:51:16.730186Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:51:16.730186Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:51:16.730186Z", "running": 0, "size": 1}}] 2022-04-09T11:51:26.930 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:51:27.660 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:51:27.660 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:51:27.560159Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:51:27.560159Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:51:27.560159Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:51:27.560159Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:51:27.560159Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:51:27.560159Z", "running": 0, "size": 1}}] 2022-04-09T11:51:37.685 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:51:38.359 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:51:38.360 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:51:38.211815Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:51:38.211815Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:51:38.211815Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:51:38.211815Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:51:38.211815Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:51:38.211815Z", "running": 0, "size": 1}}] 2022-04-09T11:51:48.384 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:51:49.081 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:51:49.082 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:51:48.954118Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:51:48.954118Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:51:48.954118Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:51:48.954118Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:51:48.954118Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:51:48.954118Z", "running": 0, "size": 1}}] 2022-04-09T11:51:59.107 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:51:59.824 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:51:59.825 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:51:59.635320Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:51:59.635320Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:51:59.635320Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:51:59.635320Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:51:59.635320Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:51:59.635320Z", "running": 0, "size": 1}}] 2022-04-09T11:52:06.470 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:52:06.466408 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:52:07.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:52:07.094783 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:52:07.085+0000 7f0d52f38380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:52:07.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:52:07.085+0000 7f0d52f38380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:52:07.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:52:07.085+0000 7f0d52f38380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:52:07.099 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:52:09.847 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:52:10.467 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:52:10.468 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:52:10.362594Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:52:10.362594Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:52:10.362594Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:52:10.362594Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:52:10.362594Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:52:10.362594Z", "running": 0, "size": 1}}] 2022-04-09T11:52:20.493 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:52:21.161 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:52:21.161 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:52:21.022882Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:52:21.022882Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:52:21.022882Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:52:21.022882Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:52:21.022882Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:52:21.022882Z", "running": 0, "size": 1}}] 2022-04-09T11:52:31.186 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:52:31.854 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:52:31.854 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:52:31.742000Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:52:31.742000Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:52:31.742000Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:52:31.742000Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:52:31.742000Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:52:31.742000Z", "running": 0, "size": 1}}] 2022-04-09T11:52:41.879 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:52:42.547 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:52:42.547 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:52:42.398112Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:52:42.398112Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:52:42.398112Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:52:42.398112Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:52:42.398112Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:52:42.398112Z", "running": 0, "size": 1}}] 2022-04-09T11:52:52.571 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:52:53.219 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:52:53.220 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:52:53.104102Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:52:53.104102Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:52:53.104102Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:52:53.104102Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:52:53.104102Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:52:53.104102Z", "running": 0, "size": 1}}] 2022-04-09T11:53:03.242 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:53:03.873 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:53:03.873 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:53:03.768373Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:53:03.768373Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:53:03.768373Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:53:03.768373Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:53:03.768373Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:53:03.768373Z", "running": 0, "size": 1}}] 2022-04-09T11:53:13.897 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:53:14.620 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:53:14.621 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:53:14.481433Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:53:14.481433Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:53:14.481433Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:53:14.481433Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:53:14.481433Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:53:14.481433Z", "running": 0, "size": 1}}] 2022-04-09T11:53:24.648 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:53:25.276 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:53:25.276 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:53:25.162417Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:53:25.162417Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:53:25.162417Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:53:25.162417Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:53:25.162417Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:53:25.162417Z", "running": 0, "size": 1}}] 2022-04-09T11:53:35.300 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:53:35.961 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:53:35.962 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:53:35.822745Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:53:35.822745Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:53:35.822745Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:53:35.822745Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:53:35.822745Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:53:35.822745Z", "running": 0, "size": 1}}] 2022-04-09T11:53:45.987 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:53:46.681 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:53:46.681 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:53:46.543256Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:53:46.543256Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:53:46.543256Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:53:46.543256Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:53:46.543256Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:53:46.543256Z", "running": 0, "size": 1}}] 2022-04-09T11:53:56.710 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:53:57.387 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:53:57.388 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:53:57.263340Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:53:57.263340Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:53:57.263340Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:53:57.263340Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:53:57.263340Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:53:57.263340Z", "running": 0, "size": 1}}] 2022-04-09T11:54:07.413 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:54:08.069 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:54:08.070 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:54:07.946060Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:54:07.946060Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:54:07.946060Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:54:07.946060Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:54:07.946060Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:54:07.946060Z", "running": 0, "size": 1}}] 2022-04-09T11:54:18.094 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:54:18.759 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:54:18.759 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:54:18.636287Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:54:18.636287Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:54:18.636287Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:54:18.636287Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:54:18.636287Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:54:18.636287Z", "running": 0, "size": 1}}] 2022-04-09T11:54:28.786 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:54:29.412 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:54:29.413 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:54:29.319153Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:54:29.319153Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:54:29.319153Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:54:29.319153Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:54:29.319153Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:54:29.319153Z", "running": 0, "size": 1}}] 2022-04-09T11:54:39.437 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:54:40.103 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:54:40.104 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:54:40.004278Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:54:40.004278Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:54:40.004278Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:54:40.004278Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:54:40.004278Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:54:40.004278Z", "running": 0, "size": 1}}] 2022-04-09T11:54:50.124 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:54:50.789 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:54:50.790 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:54:50.678262Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:54:50.678262Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:54:50.678262Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:54:50.678262Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:54:50.678262Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:54:50.678262Z", "running": 0, "size": 1}}] 2022-04-09T11:55:00.815 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:55:01.429 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:55:01.429 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:55:01.320033Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:55:01.320033Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:55:01.320033Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:55:01.320033Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:55:01.320033Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:01.320033Z", "running": 0, "size": 1}}] 2022-04-09T11:55:11.450 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:55:12.137 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:55:12.137 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:55:11.996152Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:55:11.996152Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:55:11.996152Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:55:11.996152Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:55:11.996152Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:11.996152Z", "running": 0, "size": 1}}] 2022-04-09T11:55:22.160 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:55:22.857 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:55:22.857 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:55:22.705415Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:55:22.705415Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:55:22.705415Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:55:22.705415Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:55:22.705415Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:22.705415Z", "running": 0, "size": 1}}] 2022-04-09T11:55:32.879 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:55:33.557 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:55:33.557 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:55:33.442545Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:55:33.442545Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:55:33.442545Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:55:33.442545Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:55:33.442545Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:33.442545Z", "running": 0, "size": 1}}] 2022-04-09T11:55:43.581 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:55:44.190 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:55:44.191 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:55:44.082730Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:55:44.082730Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:55:44.082730Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:55:44.082730Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:55:44.082730Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:44.082730Z", "running": 0, "size": 1}}] 2022-04-09T11:55:54.221 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:55:55.011 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:55:55.012 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:55:54.900203Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:55:54.900203Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:55:54.900203Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:55:54.900203Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:55:54.900203Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:54.900203Z", "running": 0, "size": 1}}] 2022-04-09T11:56:05.030 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:56:05.652 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:56:05.652 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:56:05.548112Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:56:05.548112Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:56:05.548112Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:56:05.548112Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:05.548112Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:05.548112Z", "running": 0, "size": 1}}] 2022-04-09T11:56:15.676 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:56:16.291 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:56:16.292 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:56:16.185185Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:56:16.185185Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:56:16.185185Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:56:16.185185Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:16.185185Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:16.185185Z", "running": 0, "size": 1}}] 2022-04-09T11:56:26.317 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:56:26.941 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:56:26.941 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:56:26.835138Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:56:26.835138Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:56:26.835138Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:56:26.835138Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:26.835138Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:26.835138Z", "running": 0, "size": 1}}] 2022-04-09T11:56:36.963 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:56:37.616 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:56:37.617 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:56:37.506576Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:56:37.506576Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:56:37.506576Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:56:37.506576Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:37.506576Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:37.506576Z", "running": 0, "size": 1}}] 2022-04-09T11:56:47.641 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:56:48.297 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:56:48.297 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:56:48.185625Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:56:48.185625Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:56:48.185625Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:56:48.185625Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:48.185625Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:48.185625Z", "running": 0, "size": 1}}] 2022-04-09T11:56:58.322 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:56:58.952 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:56:58.952 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:56:58.836074Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:56:58.836074Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:56:58.836074Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:56:58.836074Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:58.836074Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:58.836074Z", "running": 0, "size": 1}}] 2022-04-09T11:57:08.977 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:57:09.607 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:57:09.607 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:57:09.506361Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:57:09.506361Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:57:09.506361Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:57:09.506361Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:09.506361Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:09.506361Z", "running": 0, "size": 1}}] 2022-04-09T11:57:19.662 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:57:20.349 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:57:20.350 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:57:20.193262Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:57:20.193262Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:57:20.193262Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:57:20.193262Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:20.193262Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:20.193262Z", "running": 0, "size": 1}}] 2022-04-09T11:57:30.378 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:57:31.002 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:57:31.002 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:57:30.897393Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:57:30.897393Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:57:30.897393Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:57:30.897393Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:30.897393Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:30.897393Z", "running": 0, "size": 1}}] 2022-04-09T11:57:34.790 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:57:34.784020 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T11:57:35.430 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 11:57:35.424275 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:57:35.415+0000 7fc4002be380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:57:35.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:57:35.415+0000 7fc4002be380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:57:35.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T11:57:35.415+0000 7fc4002be380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:57:35.431 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:57:41.025 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:57:41.716 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:57:41.717 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:57:41.604238Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:57:41.604238Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:57:41.604238Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:57:41.604238Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:41.604238Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:41.604238Z", "running": 0, "size": 1}}] 2022-04-09T11:57:51.740 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:57:52.417 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:57:52.417 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:57:52.308478Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:57:52.308478Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:57:52.308478Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:57:52.308478Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:52.308478Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:52.308478Z", "running": 0, "size": 1}}] 2022-04-09T11:58:02.442 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:58:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:58:03.224 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:58:03.028862Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:58:03.028862Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:58:03.028862Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:58:03.028862Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:03.028862Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:03.028862Z", "running": 0, "size": 1}}] 2022-04-09T11:58:13.252 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:58:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:58:13.940 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:58:13.821799Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:58:13.821799Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:58:13.821799Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:58:13.821799Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:13.821799Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:13.821799Z", "running": 0, "size": 1}}] 2022-04-09T11:58:23.967 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:58:24.614 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:58:24.614 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:58:24.495167Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:58:24.495167Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:58:24.495167Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:58:24.495167Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:24.495167Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:24.495167Z", "running": 0, "size": 1}}] 2022-04-09T11:58:34.638 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:58:35.260 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:58:35.261 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:58:35.153940Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:58:35.153940Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:58:35.153940Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:58:35.153940Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:35.153940Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:35.153940Z", "running": 0, "size": 1}}] 2022-04-09T11:58:45.285 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:58:45.991 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:58:45.992 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:58:45.858062Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:58:45.858062Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:58:45.858062Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:58:45.858062Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:45.858062Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:45.858062Z", "running": 0, "size": 1}}] 2022-04-09T11:58:56.016 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:58:56.666 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:58:56.667 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:58:56.557572Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:58:56.557572Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:58:56.557572Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:58:56.557572Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:56.557572Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:56.557572Z", "running": 0, "size": 1}}] 2022-04-09T11:59:06.695 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:59:07.313 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:59:07.314 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:59:07.208165Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:59:07.208165Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:59:07.208165Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:59:07.208165Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:07.208165Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:07.208165Z", "running": 0, "size": 1}}] 2022-04-09T11:59:17.339 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:59:17.948 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:59:17.949 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:59:17.843452Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:59:17.843452Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:59:17.843452Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:59:17.843452Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:17.843452Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:17.843452Z", "running": 0, "size": 1}}] 2022-04-09T11:59:27.971 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:59:31.241 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:59:31.241 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:59:31.139708Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:59:31.139708Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:59:31.139708Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:59:31.139708Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:31.139708Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:31.139708Z", "running": 0, "size": 1}}] 2022-04-09T11:59:41.267 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:59:41.931 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:59:41.931 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:59:41.819864Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:59:41.819864Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:59:41.819864Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:59:41.819864Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:41.819864Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:41.819864Z", "running": 0, "size": 1}}] 2022-04-09T11:59:51.956 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T11:59:52.585 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T11:59:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T11:59:52.478462Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T11:59:52.478462Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T11:59:52.478462Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T11:59:52.478462Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:52.478462Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:52.478462Z", "running": 0, "size": 1}}] 2022-04-09T12:00:02.610 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:00:03.341 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:00:03.342 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:00:03.218974Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:00:03.218974Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:00:03.218974Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:00:03.218974Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:03.218974Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:03.218974Z", "running": 0, "size": 1}}] 2022-04-09T12:00:13.360 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:00:14.110 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:00:14.110 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:00:14.000794Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:00:14.000794Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:00:14.000794Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:00:14.000794Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:14.000794Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:14.000794Z", "running": 0, "size": 1}}] 2022-04-09T12:00:24.136 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:00:24.885 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:00:24.886 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:00:24.728358Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:00:24.728358Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:00:24.728358Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:00:24.728358Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:24.728358Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:24.728358Z", "running": 0, "size": 1}}] 2022-04-09T12:00:34.912 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:00:35.525 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:00:35.525 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:00:35.415153Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:00:35.415153Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:00:35.415153Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:00:35.415153Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:35.415153Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:35.415153Z", "running": 0, "size": 1}}] 2022-04-09T12:00:45.548 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:00:46.168 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:00:46.168 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:00:46.055638Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:00:46.055638Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:00:46.055638Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:00:46.055638Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:46.055638Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:46.055638Z", "running": 0, "size": 1}}] 2022-04-09T12:00:56.195 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:00:56.833 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:00:56.834 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:00:56.707164Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:00:56.707164Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:00:56.707164Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:00:56.707164Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:56.707164Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:56.707164Z", "running": 0, "size": 1}}] 2022-04-09T12:00:58.199 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:00:58.192886 E | ceph-rbd-mirror-controller: failed to reconcile failed to detect running and desired ceph version: failed to detect ceph image version: failed to complete ceph version job: failed to run CmdReporter ceph-rbd-mirror-controller-detect-version successfully. failed waiting for results ConfigMap ceph-rbd-mirror-controller-detect-version. timed out waiting for results ConfigMap 2022-04-09T12:00:58.199 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:00:58.192927 E | ceph-rbd-mirror-controller: failed to reconcile CephRBDMirror "rook-ceph/default-rbd-mirror". failed to detect running and desired ceph version: failed to detect ceph image version: failed to complete ceph version job: failed to run CmdReporter ceph-rbd-mirror-controller-detect-version successfully. failed waiting for results ConfigMap ceph-rbd-mirror-controller-detect-version. timed out waiting for results ConfigMap 2022-04-09T12:01:06.859 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:01:07.530 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:01:07.530 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:01:07.386561Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:01:07.386561Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:01:07.386561Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:01:07.386561Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:07.386561Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:07.386561Z", "running": 0, "size": 1}}] 2022-04-09T12:01:17.554 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:01:18.177 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:01:18.177 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:01:18.066991Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:01:18.066991Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:01:18.066991Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:01:18.066991Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:18.066991Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:18.066991Z", "running": 0, "size": 1}}] 2022-04-09T12:01:28.203 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:01:28.915 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:01:28.915 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:01:28.752484Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:01:28.752484Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:01:28.752484Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:01:28.752484Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:28.752484Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:28.752484Z", "running": 0, "size": 1}}] 2022-04-09T12:01:38.939 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:01:39.594 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:01:39.594 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:01:39.494115Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:01:39.494115Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:01:39.494115Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:01:39.494115Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:39.494115Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:39.494115Z", "running": 0, "size": 1}}] 2022-04-09T12:01:49.621 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:01:50.311 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:01:50.312 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:01:50.157857Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:01:50.157857Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:01:50.157857Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:01:50.157857Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:50.157857Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:50.157857Z", "running": 0, "size": 1}}] 2022-04-09T12:02:00.338 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:02:01.017 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:02:01.018 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:02:00.908199Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:02:00.908199Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:02:00.908199Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:02:00.908199Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:02:00.908199Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:02:00.908199Z", "running": 0, "size": 1}}] 2022-04-09T12:02:11.044 DEBUG:teuthology.orchestra.run.smithi052:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-mtmfz -- ceph orch ls -f json 2022-04-09T12:02:11.719 INFO:teuthology.orchestra.run.smithi052.stdout: 2022-04-09T12:02:11.720 INFO:teuthology.orchestra.run.smithi052.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:46:57.000000Z", "last_refresh": "2022-04-09T12:02:11.608661Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:57.000000Z", "last_refresh": "2022-04-09T12:02:11.608661Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:40:21.000000Z", "last_refresh": "2022-04-09T12:02:11.608661Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:42:23.000000Z", "last_refresh": "2022-04-09T12:02:11.608661Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:02:11.608661Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:02:11.608661Z", "running": 0, "size": 1}}] 2022-04-09T12:02:11.747 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:02:11.748 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 530, in ceph_config_keyring File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:02:11.749 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-04-09T12:02:11.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-04-09T12:02:11.766 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 478, in rook_post_config File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:02:11.767 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 442, in rook_toolbox File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:02:11.819 DEBUG:teuthology.orchestra.remote:smithi052:rook/deploy/examples/operator.yaml is 20KB 2022-04-09T12:02:11.833 DEBUG:teuthology.orchestra.run.smithi052:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-04-09T12:02:11.907 INFO:teuthology.orchestra.run.smithi052.stdout:deployment.apps "rook-ceph-tools" deleted 2022-04-09T12:02:11.925 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 379, in rook_cluster File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:02:11.926 DEBUG:teuthology.orchestra.run.smithi052:> kubectl delete -f cluster.yaml 2022-04-09T12:02:12.049 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.041247 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-04-09T12:02:12.049 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.041349 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-04-09T12:02:12.050 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.041426 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-04-09T12:02:12.050 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.041461 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-04-09T12:02:12.050 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.041474 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-04-09T12:02:12.050 INFO:tasks.rook.operator.smithi052.stdout:I0409 12:02:12.041509 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-04-09T12:02:12.051 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.041709 I | operator: successfully started the controller-runtime manager 2022-04-09T12:02:12.054 INFO:teuthology.orchestra.run.smithi052.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-04-09T12:02:12.089 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.083493 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-04-09T12:02:12.090 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.083518 I | operator: watching all namespaces for Ceph CRs 2022-04-09T12:02:12.090 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.083559 I | operator: setting up schemes 2022-04-09T12:02:12.093 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.086423 I | operator: setting up the controller-runtime manager 2022-04-09T12:02:12.297 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.290897 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-04-09T12:02:12.302 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.293855 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-09T12:02:12.302 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.293889 I | ceph-cluster-controller: successfully started 2022-04-09T12:02:12.304 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.293940 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-09T12:02:12.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.293966 I | ceph-crashcollector-controller: successfully started 2022-04-09T12:02:12.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.293996 I | ceph-block-pool-controller: successfully started 2022-04-09T12:02:12.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294028 I | ceph-object-store-user-controller: successfully started 2022-04-09T12:02:12.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294056 I | ceph-object-realm-controller: successfully started 2022-04-09T12:02:12.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294075 I | ceph-object-zonegroup-controller: successfully started 2022-04-09T12:02:12.306 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294090 I | ceph-object-zone-controller: successfully started 2022-04-09T12:02:12.306 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294206 I | ceph-object-controller: successfully started 2022-04-09T12:02:12.308 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294236 I | ceph-file-controller: successfully started 2022-04-09T12:02:12.308 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294270 I | ceph-nfs-controller: successfully started 2022-04-09T12:02:12.309 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294309 I | ceph-rbd-mirror-controller: successfully started 2022-04-09T12:02:12.309 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294349 I | ceph-client-controller: successfully started 2022-04-09T12:02:12.309 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294380 I | ceph-filesystem-mirror-controller: successfully started 2022-04-09T12:02:12.309 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294422 I | operator: rook-ceph-operator-config-controller successfully started 2022-04-09T12:02:12.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294439 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-04-09T12:02:12.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294456 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-04-09T12:02:12.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294485 I | ceph-bucket-topic: successfully started 2022-04-09T12:02:12.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294509 I | ceph-bucket-notification: successfully started 2022-04-09T12:02:12.311 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294526 I | ceph-bucket-notification: successfully started 2022-04-09T12:02:12.311 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294542 I | ceph-fs-subvolumegroup-controller: successfully started 2022-04-09T12:02:12.311 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.294551 I | blockpool-rados-namespace-controller: successfully started 2022-04-09T12:02:12.311 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.295532 I | operator: starting the controller-runtime manager 2022-04-09T12:02:12.410 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.403854 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-09T12:02:12.410 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.403912 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-09T12:02:12.411 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.403936 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-09T12:02:12.415 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.408110 I | operator: rook-ceph-operator-config-controller done reconciling 2022-04-09T12:02:12.418 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.410026 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:12.551 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.545147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:02:12.561 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:12.555863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:02:13.117 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:13.111160 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:13.099+0000 7f85c6cce380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:13.118 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:13.099+0000 7f85c6cce380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:13.118 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:13.099+0000 7f85c6cce380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:13.118 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:13.192 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:13.186724 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:13.913 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:13.907167 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:13.895+0000 7fa87b8ab380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:13.914 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:13.895+0000 7fa87b8ab380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:13.914 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:13.895+0000 7fa87b8ab380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:13.914 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:13.927 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:13.921259 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:14.542 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:14.536162 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:14.523+0000 7f07d3949380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:14.542 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:14.523+0000 7f07d3949380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:14.543 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:14.523+0000 7f07d3949380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:14.543 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:14.569 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:14.563946 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:15.189 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:15.182907 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:15.171+0000 7f05598a3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:15.189 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:15.171+0000 7f05598a3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:15.189 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:15.171+0000 7f05598a3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:15.190 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:15.262 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:15.255653 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:15.970 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:15.963924 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:15.951+0000 7f9ecd99c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:15.970 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:15.951+0000 7f9ecd99c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:15.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:15.951+0000 7f9ecd99c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:15.971 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:16.058 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:16.052552 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:16.687 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:16.680884 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:16.671+0000 7f77c62d7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:16.687 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:16.671+0000 7f77c62d7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:16.688 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:16.671+0000 7f77c62d7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:16.688 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:16.855 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:16.848975 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:17.515 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:17.509285 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:17.499+0000 7fd81d7b3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:17.516 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:17.499+0000 7fd81d7b3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:17.516 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:17.499+0000 7fd81d7b3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:17.516 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:17.842 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:17.836411 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:18.450 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:18.444351 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:18.431+0000 7fdd7e9d0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:18.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:18.431+0000 7fdd7e9d0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:18.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:18.431+0000 7fdd7e9d0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:18.451 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:19.100 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:19.094567 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:19.742 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:19.736442 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:19.727+0000 7fd8d2caa380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:19.743 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:19.727+0000 7fd8d2caa380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:19.743 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:19.727+0000 7fd8d2caa380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:19.743 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:21.032 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:21.025966 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:21.679 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:21.673190 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:21.663+0000 7fb512d8d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:21.679 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:21.663+0000 7fb512d8d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:21.680 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:21.663+0000 7fb512d8d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:21.680 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:22.658 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:22.652026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:02:22.669 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:22.663757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:02:24.244 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:24.238775 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:24.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:24.914071 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:24.903+0000 7f0bac1ab380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:24.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:24.903+0000 7f0bac1ab380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:24.921 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:24.903+0000 7f0bac1ab380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:24.921 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:30.050 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:30.044801 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:30.840 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:30.834281 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:30.823+0000 7f04d35ae380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:30.841 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:30.823+0000 7f04d35ae380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:30.841 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:30.823+0000 7f04d35ae380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:30.841 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:32.735 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:32.729596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:02:32.746 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:32.740978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:02:41.091 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:41.085196 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:02:41.723 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:41.716888 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:02:41.707+0000 7f8a2cbf3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:02:41.724 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:41.707+0000 7f8a2cbf3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:02:41.724 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:02:41.707+0000 7f8a2cbf3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:02:41.725 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:02:43.262 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:43.256387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:02:43.273 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:43.267333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:02:53.360 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:53.353864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:02:53.368 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:02:53.362554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:02.214 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:02.208111 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:03:02.821 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:02.814341 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:03:02.806+0000 7f77f791b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:03:02.821 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:03:02.806+0000 7f77f791b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:03:02.821 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:03:02.806+0000 7f77f791b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:03:02.822 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:03:03.428 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:03.421247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:03.468 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:03.462274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:13.541 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:13.534849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:13.549 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:13.543902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:23.645 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:23.638592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:23.659 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:23.652568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:33.757 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:33.750587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:33.805 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:33.798585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:43.789 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:43.783579 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:03:43.875 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:43.869062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:43.922 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:43.916399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:44.419 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:44.413282 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:03:44.406+0000 7fc6c7900380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:03:44.420 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:03:44.406+0000 7fc6c7900380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:03:44.420 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:03:44.406+0000 7fc6c7900380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:03:44.420 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:03:54.005 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:53.999573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:03:54.018 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:03:54.010620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:04.121 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:04.115280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:04.134 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:04.128514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:14.200 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:14.194348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:14.213 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:14.206839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:24.307 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:24.300361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:24.320 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:24.314092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:34.418 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:34.412207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:34.429 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:34.422688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:44.546 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:44.539612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:44.557 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:44.551062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:54.637 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:54.631197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:04:54.648 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:04:54.642128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:04.769 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:04.763193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:04.781 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:04.774980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:06.350 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:06.343991 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:05:06.973 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:06.966892 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:05:06.957+0000 7fe9aa521380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:05:06.974 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:05:06.957+0000 7fe9aa521380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:05:06.974 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:05:06.957+0000 7fe9aa521380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:05:06.974 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:05:14.903 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:14.897396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:14.916 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:14.910084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:25.009 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:25.003130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:25.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:25.013700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:35.117 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:35.110485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:35.128 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:35.121287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:45.234 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:45.227396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:45.244 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:45.238459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:55.359 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:55.352857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:05:55.370 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:05:55.363921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:05.443 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:05.437047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:05.455 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:05.448741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:15.586 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:15.579237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:15.596 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:15.590333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:25.680 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:25.674261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:25.692 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:25.685823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:35.813 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:35.806816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:35.824 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:35.817691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:45.937 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:45.931025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:45.949 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:45.943691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:56.049 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:56.042570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:06:56.060 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:06:56.054475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:06.163 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:06.157044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:06.176 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:06.170365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:16.269 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:16.263320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:16.280 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:16.274901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:26.365 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:26.358976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:26.376 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:26.371261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:36.449 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:36.443143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:36.459 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:36.453927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:46.548 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:46.542760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:46.559 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:46.554003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:50.818 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:50.812180 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:07:51.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:51.445383 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:07:51.435+0000 7efde1fbf380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:07:51.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:07:51.435+0000 7efde1fbf380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:07:51.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:07:51.435+0000 7efde1fbf380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:07:51.452 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:07:56.666 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:56.660397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:07:56.677 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:07:56.671962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:06.755 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:06.749159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:06.766 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:06.761438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:16.837 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:16.832315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:16.847 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:16.842093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:26.956 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:26.950277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:26.966 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:26.961653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:37.061 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:37.055764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:37.074 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:37.068915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:47.199 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:47.194056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:47.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:47.228451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:57.307 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:57.302425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:08:57.318 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:08:57.313160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:07.397 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:07.392485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:07.408 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:07.403579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:17.514 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:17.509551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:17.525 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:17.521094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:27.600 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:27.595396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:27.611 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:27.606627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:37.737 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:37.732025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:37.748 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:37.743223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:47.807 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:47.802629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:47.818 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:47.813258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:57.916 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:57.911710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:09:57.928 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:09:57.923951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:08.059 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:08.054008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:08.070 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:08.066256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:18.149 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:18.144632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:18.160 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:18.155458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:28.258 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:28.253322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:28.269 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:28.264462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:38.369 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:38.364473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:38.380 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:38.376430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:48.495 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:48.490303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:48.505 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:48.501115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:58.616 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:58.612041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:10:58.630 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:10:58.625109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:08.770 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:08.765328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:08.784 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:08.779288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:18.889 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:18.884345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:18.901 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:18.897307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:28.965 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:28.961015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:28.978 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:28.974025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:39.050 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:39.045892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:39.061 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:39.056928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:49.151 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:49.146472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:49.161 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:49.157439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:59.360 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:59.355770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:11:59.372 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:11:59.368219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:09.471 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:09.466884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:09.483 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:09.478704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:19.689 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:19.684549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:19.698 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:19.693860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:30.077 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:30.072525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:30.088 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:30.084115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:40.154 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:40.146867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:40.196 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:40.191713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:50.267 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:50.262634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:12:50.279 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:12:50.274427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:00.373 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:00.368836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:00.384 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:00.379776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:10.494 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:10.489593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:10.505 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:10.501338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:19.136 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:19.131935 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:13:19.872 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:19.867092 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:13:19.856+0000 7fa0d3dfd380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:13:19.872 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:13:19.856+0000 7fa0d3dfd380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:13:19.873 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:13:19.856+0000 7fa0d3dfd380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:13:19.873 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:13:20.603 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:20.598647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:20.615 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:20.611179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:30.719 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:30.715181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:30.728 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:30.724175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:40.848 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:40.843734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:40.859 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:40.855086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:50.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:50.949795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:13:50.964 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:13:50.960226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:01.032 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:01.027465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:01.042 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:01.037977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:11.140 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:11.136238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:11.166 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:11.162310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:21.275 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:21.270801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:21.286 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:21.282044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:31.404 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:31.399553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:31.415 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:31.411797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:41.536 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:41.531818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:41.547 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:41.542843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:51.671 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:51.666782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:14:51.681 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:14:51.677079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:01.757 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:01.752713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:01.765 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:01.761629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:11.898 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:11.893507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:11.911 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:11.907123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:21.973 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:21.969437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:21.983 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:21.979487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:32.041 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:32.036992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:32.106 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:32.102237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:42.276 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:42.272007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:42.286 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:42.282683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:52.385 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:52.381043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:15:52.397 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:15:52.394252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:02.464 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:02.460856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:02.477 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:02.473373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:12.589 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:12.585529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:12.601 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:12.597955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:22.674 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:22.670142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:22.684 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:22.681414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:32.776 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:32.773200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:32.787 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:32.783992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:42.898 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:42.894431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:42.909 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:42.906043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:53.075 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:53.070689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:16:53.085 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:16:53.081640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:03.185 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:03.181754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:03.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:03.192095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:13.278 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:13.275064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:13.288 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:13.285162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:23.515 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:23.511743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:23.527 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:23.523694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:33.702 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:33.699065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:33.712 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:33.709441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:43.812 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:43.808895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:43.823 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:43.820234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:53.891 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:53.887671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:17:53.900 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:17:53.897257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:04.003 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:03.999808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:04.016 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:04.012956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:14.096 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:14.092939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:14.154 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:14.151554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:24.238 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:24.234876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:24.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:24.246518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:34.312 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:34.309070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:34.323 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:34.320454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:44.395 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:44.391930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:44.407 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:44.404214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:54.567 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:54.563844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:18:54.577 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:18:54.574341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:04.635 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:04.629855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:04.677 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:04.674619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:14.741 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:14.738276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:14.818 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:14.815514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:24.880 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:24.877368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:24.933 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:24.930132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:34.998 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:34.995321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:35.006 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:35.004014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:45.071 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:45.068618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:45.079 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:45.077299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:55.159 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:55.157150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:19:55.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:19:55.204982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:05.268 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:05.265730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:05.279 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:05.276412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:15.371 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:15.367799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:15.388 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:15.385464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:25.496 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:25.493815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:25.507 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:25.504813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:35.689 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:35.687114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:35.703 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:35.700366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:45.765 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:45.762104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:45.774 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:45.771972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:55.865 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:55.862535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:20:55.877 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:20:55.874430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:05.940 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:05.937767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:05.989 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:05.987181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:16.138 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:16.135567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:16.150 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:16.147323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:26.209 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:26.205852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:26.303 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:26.294455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:36.370 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:36.368129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:36.382 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:36.379936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:46.490 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:46.487440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:46.500 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:46.497349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:56.569 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:56.567010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:21:56.609 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:21:56.607210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:06.709 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:06.707249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:06.721 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:06.718883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:16.802 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:16.800162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:16.812 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:16.810578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:26.887 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:26.885482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:26.898 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:26.896336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:37.016 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:37.014188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:37.027 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:37.025250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:47.089 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:47.087347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:47.112 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:47.109829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:57.183 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:57.181495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:22:57.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:22:57.193247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:07.308 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:07.306462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:07.319 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:07.317302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:17.585 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:17.583323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:17.595 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:17.593350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:27.699 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:27.696891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:27.709 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:27.707251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:37.776 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:37.774510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:37.827 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:37.824867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:47.894 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:47.892419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:47.905 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:47.903037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:57.970 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:57.967631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:23:58.017 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:23:58.015555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:08.088 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:08.085829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:08.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:08.096907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:15.263 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:15.261462 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:24:15.919 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:15.917513 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:24:15.910+0000 7f0e7bdcb380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:24:15.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:24:15.910+0000 7f0e7bdcb380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:24:15.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:24:15.910+0000 7f0e7bdcb380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:24:15.920 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:24:18.197 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:18.195040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:18.209 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:18.207405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:28.354 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:28.351733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:28.365 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:28.363661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:38.479 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:38.477328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:38.489 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:38.487507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:48.586 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:48.583872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:48.597 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:48.595051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:58.697 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:58.694721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:24:58.707 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:24:58.705010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:08.780 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:08.778038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:08.819 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:08.817643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:18.933 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:18.931200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:18.944 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:18.943080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:29.008 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:29.006016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:29.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:29.017004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:39.112 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:39.110183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:39.125 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:39.123487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:49.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:49.193099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:49.243 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:49.241369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:59.338 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:59.336410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:25:59.354 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:25:59.351211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:09.419 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:09.417421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:09.464 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:09.462553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:19.552 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:19.550759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:19.564 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:19.562468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:29.631 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:29.629097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:29.641 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:29.639890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:39.743 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:39.741779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:39.754 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:39.752502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:49.822 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:49.820215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:49.830 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:49.829151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:59.901 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:59.899769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:26:59.912 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:26:59.910720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:10.000 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:09.998137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:10.012 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:10.010482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:20.147 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:20.144779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:20.156 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:20.154767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:30.263 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:30.261770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:30.276 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:30.274278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:40.381 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:40.379257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:40.392 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:40.391116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:50.472 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:50.470339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:27:50.554 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:27:50.552088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:00.665 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:00.663878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:00.674 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:00.671940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:10.742 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:10.740180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:10.754 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:10.752571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:20.835 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:20.833303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:20.883 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:20.880898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:30.996 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:30.994113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:31.005 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:31.003921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:41.139 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:41.136856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:41.151 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:41.150291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:51.254 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:51.252377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:28:51.265 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:28:51.262833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:01.362 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:01.360309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:01.373 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:01.371913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:11.611 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:11.609953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:11.622 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:11.620781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:21.881 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:21.879104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:21.892 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:21.890156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:31.969 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:31.966868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:31.978 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:31.977158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:42.035 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:42.033931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:42.047 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:42.043469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:52.106 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:52.104558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:29:52.118 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:29:52.116988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:02.179 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:02.177954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:02.188 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:02.187109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:12.253 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:12.251667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:12.265 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:12.263670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:22.332 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:22.330540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:22.345 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:22.343450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:32.458 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:32.456200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:32.469 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:32.468245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:42.587 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:42.584956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:42.599 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:42.598139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:52.710 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:52.708761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:30:52.720 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:30:52.719114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:02.833 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:02.831069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:02.841 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:02.839908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:12.965 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:12.963567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:12.976 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:12.975132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:23.082 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:23.080180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:23.093 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:23.091668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:33.188 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:33.186233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:33.199 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:33.197973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:43.286 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:43.284491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:43.298 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:43.296459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:53.403 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:53.401506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:31:53.436 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:31:53.434565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:03.536 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:03.534898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:03.547 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:03.545881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:13.636 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:13.634112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:13.646 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:13.644950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:23.771 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:23.769191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:23.781 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:23.779439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:33.921 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:33.920241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:33.932 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:33.931130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:44.023 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:44.020888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:44.034 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:44.032942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:54.095 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:54.093458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:32:54.105 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:32:54.103785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:04.212 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:04.210667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:04.220 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:04.219147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:14.292 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:14.290409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:14.301 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:14.299973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:24.393 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:24.389795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:24.403 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:24.400875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:34.495 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:34.493593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:34.504 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:34.503361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:44.568 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:44.566477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:44.577 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:44.576225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:54.796 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:54.794316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:33:54.807 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:33:54.806017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:04.892 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:04.891089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:04.902 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:04.901053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:14.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:14.969832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:14.979 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:14.977532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:25.046 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:25.044431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:25.055 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:25.053984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:35.315 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:35.313812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:35.326 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:35.325263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:45.433 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:45.431098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:45.445 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:45.443578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:55.504 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:55.502254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:34:55.513 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:34:55.512061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:05.624 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:05.622761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:05.637 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:05.635485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:15.724 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:15.722163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:15.738 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:15.736774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:25.810 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:25.808862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:25.822 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:25.820859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:35.933 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:35.931348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:35.944 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:35.942865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:46.006 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:46.004084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:46.017 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:46.014849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:56.116 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:56.114678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:35:56.126 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:35:56.125224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:06.198 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:06.196857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:06.211 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:06.209335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:16.314 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:16.311848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:16.326 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:16.325224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:26.421 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:26.419001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:26.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:26.429923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:36.533 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:36.531609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:36.544 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:36.543301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:46.620 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:46.618269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:46.630 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:46.628775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:56.750 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:56.748325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:36:56.760 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:36:56.759135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:06.820 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:06.818356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:06.830 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:06.828994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:16.946 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:16.944939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:16.956 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:16.954771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:27.030 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:27.028659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:27.040 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:27.038295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:37.156 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:37.154964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:37.169 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:37.167597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:47.299 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:47.297868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:47.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:47.309296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:57.388 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:57.386071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:37:57.401 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:37:57.399648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:07.500 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:07.499213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:07.513 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:07.511458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:17.584 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:17.582796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:17.595 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:17.593624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:27.701 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:27.699285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:27.712 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:27.710819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:37.836 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:37.833979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:37.848 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:37.847180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:47.969 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:47.967019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:47.982 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:47.980535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:58.067 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:58.064803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:38:58.078 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:38:58.076478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:08.152 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:08.150377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:08.165 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:08.163781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:18.295 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:18.294004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:18.308 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:18.307108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:28.443 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:28.441526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:28.454 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:28.452115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:38.521 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:38.519617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:38.532 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:38.530160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:48.613 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:48.611916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:48.668 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:48.666502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:58.812 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:58.810019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:39:58.823 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:39:58.821368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:08.884 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:08.882289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:08.932 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:08.930205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:18.994 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:18.992072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:19.005 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:19.003603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:29.174 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:29.170736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:29.186 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:29.184593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:39.254 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:39.252282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:39.265 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:39.263862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:49.360 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:49.357918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:49.372 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:49.370105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:55.931 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:55.925891 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:40:56.551 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:56.549276 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:40:56.542+0000 7f0b2f0cc380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:40:56.552 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:40:56.542+0000 7f0b2f0cc380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:40:56.552 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:40:56.542+0000 7f0b2f0cc380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:40:56.552 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:40:59.441 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:59.439264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:40:59.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:40:59.450214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:09.567 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:09.565650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:09.579 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:09.577523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:19.642 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:19.639213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:19.652 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:19.649296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:29.753 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:29.751603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:29.763 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:29.761903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:39.900 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:39.898022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:39.911 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:39.909418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:49.979 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:49.977167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:41:49.992 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:41:49.990818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:00.138 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:00.136348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:00.151 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:00.149525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:10.258 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:10.256506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:10.270 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:10.268887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:20.343 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:20.341208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:20.355 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:20.352678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:30.426 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:30.423824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:30.471 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:30.469039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:40.574 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:40.572436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:40.587 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:40.585073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:50.694 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:50.692329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:42:50.706 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:42:50.704336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:00.816 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:00.814335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:00.835 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:00.824716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:10.946 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:10.943909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:10.956 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:10.953538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:21.030 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:21.027552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:21.042 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:21.040004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:31.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:31.193069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:31.209 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:31.206914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:41.332 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:41.330511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:41.341 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:41.339233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:51.417 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:51.412887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:43:51.427 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:43:51.425715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:01.495 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:01.492825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:01.505 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:01.503517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:11.597 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:11.595360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:11.607 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:11.605624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:21.667 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:21.664741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:21.678 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:21.676555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:31.749 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:31.747526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:31.761 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:31.759768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:41.861 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:41.859539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:41.872 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:41.870323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:51.992 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:51.989851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:44:52.002 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:44:52.000760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:02.118 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:02.115863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:02.127 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:02.125108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:12.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:12.239685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:12.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:12.249063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:22.380 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:22.377525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:22.390 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:22.387745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:32.450 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:32.447489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:32.461 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:32.459388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:42.534 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:42.532347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:42.545 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:42.542681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:52.646 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:52.643619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:45:52.659 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:45:52.657108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:02.765 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:02.762772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:02.776 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:02.773352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:12.835 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:12.832376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:12.845 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:12.843121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:22.916 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:22.913711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:22.933 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:22.931513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:33.074 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:33.071893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:33.085 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:33.083467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:43.153 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:43.150992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:43.166 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:43.163478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:53.247 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:53.244408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:46:53.258 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:46:53.255871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:03.355 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:03.353053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:03.366 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:03.364362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:13.441 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:13.438601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:13.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:13.449454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:23.560 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:23.557781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:23.572 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:23.569813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:33.681 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:33.674217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:33.690 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:33.687584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:43.787 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:43.784748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:43.797 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:43.794485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:53.862 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:53.859858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:47:53.874 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:47:53.871901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:03.966 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:03.963466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:03.977 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:03.974446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:14.103 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:14.100752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:14.116 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:14.114557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:24.186 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:24.183310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:24.197 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:24.195004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:34.294 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:34.291077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:34.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:34.303021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:44.374 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:44.371409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:44.384 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:44.382119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:54.458 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:54.455685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:48:54.471 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:48:54.469107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:04.572 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:04.569681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:04.586 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:04.583689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:14.709 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:14.706231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:14.719 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:14.716644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:24.830 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:24.827719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:24.840 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:24.837485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:34.955 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:34.952174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:34.965 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:34.962883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:45.072 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:45.069245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:45.083 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:45.080427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:55.158 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:55.155283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:49:55.170 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:49:55.167475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:05.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:05.269350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:05.284 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:05.281999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:15.387 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:15.384643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:15.401 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:15.397898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:25.477 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:25.474658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:25.489 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:25.486390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:35.590 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:35.587076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:35.602 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:35.599571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:45.715 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:45.712002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:45.728 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:45.725165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:55.832 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:55.829008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:50:55.866 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:50:55.863699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:05.945 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:05.941238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:05.955 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:05.952599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:16.027 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:16.024247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:16.079 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:16.076705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:26.170 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:26.167089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:26.226 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:26.223927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:36.292 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:36.289027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:36.302 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:36.299533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:46.366 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:46.363253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:46.378 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:46.375754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:56.443 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:56.440534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:51:56.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:51:56.449146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:06.563 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:06.559840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:06.575 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:06.572701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:16.635 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:16.632215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:16.646 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:16.643263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:26.720 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:26.717510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:26.731 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:26.729155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:36.815 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:36.811908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:36.827 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:36.823906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:46.935 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:46.931955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:46.945 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:46.942824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:57.095 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:57.092492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:52:57.106 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:52:57.103859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:07.199 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:07.196704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:07.213 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:07.210563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:17.280 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:17.276851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:17.291 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:17.288497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:27.362 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:27.359239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:27.370 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:27.367106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:37.448 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:37.444631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:37.457 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:37.454399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:47.518 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:47.511941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:47.526 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:47.523523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:57.623 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:57.620279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:53:57.635 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:53:57.631461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:07.796 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:07.793287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:07.807 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:07.804727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:17.936 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:17.932723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:17.943 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:17.940942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:28.102 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:28.099013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:28.112 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:28.109346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:38.226 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:38.223009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:38.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:38.238252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:48.304 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:48.300749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:48.313 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:48.310637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:58.409 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:58.405507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:54:58.421 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:54:58.418090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:08.520 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:08.516943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:08.532 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:08.529611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:18.660 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:18.656877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:18.674 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:18.671024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:28.737 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:28.734171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:28.750 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:28.747090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:38.848 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:38.845584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:38.859 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:38.856335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:48.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:48.948014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:48.964 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:48.961497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:59.032 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:59.028917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:55:59.044 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:55:59.040985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:09.109 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:09.106123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:09.148 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:09.145714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:19.256 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:19.252995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:19.267 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:19.264464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:29.345 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:29.342048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:29.356 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:29.353258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:39.482 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:39.479366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:39.496 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:39.490483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:49.561 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:49.558467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:49.626 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:49.622781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:59.746 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:59.743403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:56:59.756 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:56:59.753430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:09.984 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:09.980771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:09.994 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:09.991381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:20.243 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:20.239815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:20.253 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:20.250395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:30.332 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:30.328819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:30.343 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:30.340045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:36.560 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:36.557301 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T12:57:37.184 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:37.180706 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:57:37.172+0000 7f0738700380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:57:37.185 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:57:37.172+0000 7f0738700380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:57:37.185 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T12:57:37.172+0000 7f0738700380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:57:37.186 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:57:40.461 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:40.458145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:40.473 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:40.470456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:50.541 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:50.538389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:57:50.552 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:57:50.549247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:00.667 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:00.663685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:00.676 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:00.673644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:10.764 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:10.761326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:10.775 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:10.772124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:20.888 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:20.880531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:20.898 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:20.895258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:30.999 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:30.996107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:31.009 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:31.006706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:41.121 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:41.117601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:41.128 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:41.125370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:51.240 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:51.237022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:58:51.251 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:58:51.248126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:01.343 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:01.339792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:01.354 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:01.351715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:11.432 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:11.429129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:11.515 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:11.512195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:21.592 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:21.588871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:21.639 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:21.635736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:31.710 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:31.707666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:31.721 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:31.718847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:41.793 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:41.790308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:41.803 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:41.800610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:51.902 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:51.899252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T12:59:51.915 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 12:59:51.911219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:02.020 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:02.016597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:02.033 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:02.030427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:12.125 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:12.121932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:12.134 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:12.131945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:22.247 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:22.244542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:22.258 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:22.255161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:32.380 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:32.376686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:32.416 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:32.412770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:42.500 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:42.497779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:42.511 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:42.508303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:52.686 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:52.683362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:00:52.714 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:00:52.711496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:02.861 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:02.858379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:02.876 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:02.873520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:13.036 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:13.033540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:13.047 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:13.044581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:23.134 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:23.131224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:23.144 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:23.141220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:33.208 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:33.205173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:33.255 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:33.252158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:43.378 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:43.375663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:43.389 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:43.386178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:53.476 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:53.473495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:01:53.489 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:01:53.486753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:03.597 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:03.594210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:03.608 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:03.605460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:13.678 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:13.675337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:13.688 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:13.685250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:23.752 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:23.749211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:23.761 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:23.759075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:33.861 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:33.858499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:33.874 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:33.871874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:43.976 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:43.973485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:43.987 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:43.985131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:54.106 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:54.103730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:02:54.116 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:02:54.113795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:04.188 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:04.185508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:04.200 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:04.197270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:14.260 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:14.257763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:14.270 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:14.267519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:24.351 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:24.348647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:24.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:24.360219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:34.459 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:34.456016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:34.469 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:34.466719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:44.612 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:44.608879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:44.623 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:44.620715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:54.690 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:54.686947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:03:54.700 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:03:54.697564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:04.769 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:04.766113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:04.780 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:04.777489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:14.850 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:14.847153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:14.917 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:14.914235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:25.048 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:25.045504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:25.058 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:25.055423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:35.153 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:35.149941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:35.164 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:35.161905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:45.259 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:45.256738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:45.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:45.269095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:55.362 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:55.359769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:04:55.374 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:04:55.371124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:05.490 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:05.487277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:05.501 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:05.497893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:15.608 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:15.605679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:15.619 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:15.616881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:25.721 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:25.716595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:25.731 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:25.728918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:35.802 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:35.799316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:35.846 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:35.843432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:45.949 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:45.945788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:45.959 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:45.956704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:56.065 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:56.061782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:05:56.081 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:05:56.078955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:06.214 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:06.211055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:06.223 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:06.221113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:16.304 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:16.300913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:16.361 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:16.357831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:26.457 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:26.454744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:26.469 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:26.466339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:36.541 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:36.538727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:36.552 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:36.549317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:46.632 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:46.629358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:46.691 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:46.688047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:56.756 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:56.753634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:06:56.767 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:06:56.764916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:06.839 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:06.836045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:06.852 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:06.849422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:17.022 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:17.019188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:17.033 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:17.030496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:27.116 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:27.112992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:27.126 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:27.122965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:37.199 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:37.196054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:37.258 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:37.255122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:47.334 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:47.331188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:47.342 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:47.339946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:57.450 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:57.447399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:07:57.462 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:07:57.459907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:07.566 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:07.562863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:07.579 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:07.573665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:17.651 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:17.648082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:17.690 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:17.687517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:27.795 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:27.791681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:27.806 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:27.803629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:37.887 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:37.884412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:37.898 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:37.895596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:47.973 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:47.970201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:47.983 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:47.981000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:58.065 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:58.061907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:08:58.075 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:08:58.073013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:08.189 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:08.186460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:08.200 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:08.197909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:18.313 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:18.310193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:18.322 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:18.319411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:28.407 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:28.404559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:28.419 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:28.416690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:38.584 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:38.581704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:38.595 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:38.592851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:48.691 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:48.688466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:48.702 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:48.699283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:58.892 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:58.889703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:09:58.904 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:09:58.901242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:09.045 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:09.042725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:09.055 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:09.053107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:19.170 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:19.167226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:19.182 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:19.179566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:29.262 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:29.259236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:29.312 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:29.309801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:39.383 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:39.379809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:39.393 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:39.390699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:49.524 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:49.521486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:49.533 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:49.530457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:59.637 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:59.632522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:10:59.702 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:10:59.697889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:09.763 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:09.760622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:09.774 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:09.771185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:19.881 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:19.877616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:19.890 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:19.887398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:29.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:29.967901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:30.018 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:30.015932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:40.090 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:40.087260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:40.102 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:40.099368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:50.232 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:50.229641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:11:50.243 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:11:50.240826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:00.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:00.304434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:00.321 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:00.318165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:10.388 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:10.385516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:10.400 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:10.397623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:20.471 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:20.468415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:20.482 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:20.479267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:30.654 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:30.650503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:30.665 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:30.662476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:40.735 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:40.731882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:40.745 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:40.741874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:50.808 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:50.805235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:12:50.817 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:12:50.814717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:00.878 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:00.875694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:00.921 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:00.917895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:11.017 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:11.014471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:11.037 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:11.034328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:21.115 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:21.111853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:21.131 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:21.123787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:31.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:31.237978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:31.252 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:31.249152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:41.379 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:41.375930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:41.391 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:41.388590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:51.512 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:51.508832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:13:51.528 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:13:51.525365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:01.714 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:01.710837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:01.726 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:01.723527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:11.853 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:11.849847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:11.863 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:11.860916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:17.193 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:17.190343 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T13:14:17.811 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:17.808237 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:14:17.799+0000 7f4625f26380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:14:17.812 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T13:14:17.799+0000 7f4625f26380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:14:17.812 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T13:14:17.799+0000 7f4625f26380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:14:17.812 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:14:21.932 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:21.929124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:21.990 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:21.987100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:32.084 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:32.081356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:32.094 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:32.091665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:42.229 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:42.226100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:42.238 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:42.236072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:52.322 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:52.319392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:14:52.334 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:14:52.331173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:02.421 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:02.418026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:02.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:02.427820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:12.531 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:12.527563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:12.541 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:12.538067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:22.664 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:22.660668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:22.675 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:22.672379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:32.792 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:32.788567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:32.804 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:32.801330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:42.916 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:42.912711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:42.928 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:42.924944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:52.994 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:52.991083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:15:53.005 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:15:53.002544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:03.102 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:03.099056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:03.115 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:03.112342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:13.214 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:13.211339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:13.225 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:13.222713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:23.287 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:23.284581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:23.298 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:23.296045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:33.370 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:33.367904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:33.502 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:33.379877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:43.457 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:43.454642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:43.508 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:43.504880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:53.585 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:53.582098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:16:53.664 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:16:53.660865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:03.893 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:03.890513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:03.903 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:03.900726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:13.989 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:13.985942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:13.999 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:13.996854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:24.122 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:24.119243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:24.132 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:24.129331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:34.197 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:34.194013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:34.238 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:34.235502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:44.356 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:44.352345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:44.368 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:44.365288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:54.491 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:54.488383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:17:54.501 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:17:54.497931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:04.578 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:04.574713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:04.627 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:04.624673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:14.707 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:14.703600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:14.717 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:14.714021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:24.812 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:24.809366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:24.827 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:24.824097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:34.922 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:34.918682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:34.935 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:34.931424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:45.005 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:45.002323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:45.015 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:45.012839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:55.098 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:55.095298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:18:55.110 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:18:55.107404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:05.229 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:05.226045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:05.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:05.238677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:15.345 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:15.342079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:15.356 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:15.353518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:25.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:25.427650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:25.442 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:25.439632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:35.569 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:35.566647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:35.579 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:35.576524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:45.708 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:45.704828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:45.716 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:45.713173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:55.849 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:55.845950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:19:55.860 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:19:55.857896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:05.977 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:05.974189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:05.987 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:05.984755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:16.082 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:16.078730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:16.093 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:16.090180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:26.218 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:26.215533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:26.230 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:26.227042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:36.343 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:36.340073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:36.355 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:36.352564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:46.433 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:46.430219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:46.445 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:46.442110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:56.580 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:56.577489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:20:56.591 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:20:56.588058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:06.700 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:06.697292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:06.711 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:06.708597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:16.823 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:16.819925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:16.832 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:16.829642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:26.908 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:26.905184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:26.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:26.917007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:37.011 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:37.008533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:37.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:37.021533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:47.163 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:47.159906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:47.176 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:47.173280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:57.282 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:57.279103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:21:57.293 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:21:57.290324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:07.403 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:07.400619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:07.413 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:07.410674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:17.486 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:17.482998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:17.494 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:17.491783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:27.637 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:27.633999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:27.647 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:27.644610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:37.714 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:37.710783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:37.722 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:37.719145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:47.825 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:47.821812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:47.837 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:47.833291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:57.907 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:57.904231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:22:57.918 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:22:57.915496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:07.983 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:07.980528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:07.994 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:07.991829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:18.295 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:18.291582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:18.361 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:18.357602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:28.498 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:28.495196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:28.508 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:28.505779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:38.623 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:38.620175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:38.634 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:38.631266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:48.730 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:48.727506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:49.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:48.740151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:58.810 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:58.807351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:23:58.821 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:23:58.818551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:08.926 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:08.922976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:08.937 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:08.934060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:18.999 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:18.996337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:19.010 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:19.006867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:29.101 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:29.097998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:29.111 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:29.107745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:39.172 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:39.169353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:39.183 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:39.179624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:49.260 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:49.257590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:49.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:49.268612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:59.339 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:59.336253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:24:59.347 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:24:59.344524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:09.433 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:09.429927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:09.443 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:09.440880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:19.512 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:19.508745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:19.526 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:19.523165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:29.659 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:29.656290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:29.669 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:29.666767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:39.741 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:39.737874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:39.753 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:39.750347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:49.855 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:49.852123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:25:49.867 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:25:49.864078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:00.005 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:00.001988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:00.021 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:00.018273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:10.136 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:10.132917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:10.150 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:10.147787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:20.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:20.245764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:20.262 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:20.259011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:30.328 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:30.324885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:30.339 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:30.336163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:40.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:40.427686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:40.442 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:40.439772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:50.562 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:50.559201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:26:50.573 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:26:50.570802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:00.667 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:00.664265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:00.677 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:00.674083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:10.791 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:10.788220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:10.802 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:10.799743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:20.874 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:20.871913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:20.924 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:20.921277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:31.028 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:31.025276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:31.041 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:31.038268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:41.147 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:41.142901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:41.176 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:41.154520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:51.262 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:51.258919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:27:51.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:27:51.269641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:01.366 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:01.363405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:01.377 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:01.374545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:11.491 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:11.488439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:11.502 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:11.499258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:21.576 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:21.573442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:21.588 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:21.585293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:31.887 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:31.884418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:31.901 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:31.898841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:42.003 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:42.000820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:42.015 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:42.012507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:52.149 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:52.146127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:28:52.160 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:28:52.157149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:02.261 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:02.258627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:02.273 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:02.270661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:12.385 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:12.381937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:12.393 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:12.391042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:22.460 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:22.457175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:22.471 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:22.468152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:32.595 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:32.591803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:32.606 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:32.603529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:42.741 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:42.738523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:42.752 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:42.749064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:52.840 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:52.837136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:29:52.851 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:29:52.847988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:02.940 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:02.937820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:02.950 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:02.947969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:13.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:13.015894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:13.031 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:13.028208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:23.103 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:23.100855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:23.116 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:23.113350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:33.253 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:33.250324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:33.264 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:33.261300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:43.361 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:43.358780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:43.372 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:43.370056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:53.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:53.449302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:53.464 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:53.461983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:30:57.824 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:57.821317 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T13:30:58.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:30:58.428081 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:30:58.418+0000 7f7dbb0ab380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:30:58.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T13:30:58.418+0000 7f7dbb0ab380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:30:58.432 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T13:30:58.418+0000 7f7dbb0ab380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:30:58.432 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:31:03.604 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:03.602064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:03.616 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:03.613100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:13.707 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:13.704255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:13.716 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:13.713123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:23.782 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:23.779417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:23.793 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:23.791039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:33.872 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:33.869317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:33.941 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:33.938606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:44.006 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:44.003738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:44.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:44.016160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:54.075 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:54.070826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:31:54.085 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:31:54.082631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:04.201 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:04.198184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:04.210 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:04.207957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:14.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:14.307649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:14.322 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:14.319641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:24.442 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:24.439775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:24.453 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:24.450680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:34.561 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:34.558253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:34.571 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:34.568238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:44.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:44.729054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:44.743 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:44.740631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:54.854 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:54.851666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:32:54.863 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:32:54.860785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:04.939 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:04.935999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:04.949 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:04.946505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:15.025 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:15.023095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:15.037 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:15.034787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:25.123 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:25.120549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:25.135 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:25.132737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:35.229 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:35.225988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:35.240 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:35.238136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:45.327 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:45.324764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:45.340 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:45.337908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:55.438 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:55.435685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:33:55.450 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:33:55.448252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:05.566 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:05.563553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:05.577 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:05.574739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:15.643 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:15.641145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:15.694 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:15.691975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:25.759 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:25.756544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:25.771 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:25.768493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:35.877 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:35.874737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:35.888 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:35.885803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:46.116 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:46.113922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:46.126 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:46.123891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:56.255 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:56.252421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:34:56.267 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:34:56.264503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:06.329 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:06.324017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:06.337 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:06.334519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:16.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:16.448908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:16.465 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:16.459040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:26.550 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:26.547264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:26.562 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:26.559705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:36.635 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:36.633138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:36.648 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:36.646082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:46.722 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:46.719360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:46.733 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:46.730750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:56.839 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:56.836452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:35:56.851 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:35:56.848686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:06.923 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:06.915755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:06.930 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:06.927883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:16.995 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:16.990836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:17.004 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:17.002372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:27.100 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:27.097685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:27.111 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:27.108929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:37.211 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:37.204034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:37.224 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:37.221790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:47.315 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:47.312296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:47.325 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:47.323051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:57.396 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:57.393844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:36:57.507 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:36:57.504482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:07.580 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:07.577636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:07.592 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:07.589976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:17.701 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:17.698372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:17.712 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:17.709671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:27.800 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:27.797018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:27.839 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:27.836737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:37.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:37.968552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:37.983 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:37.980499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:48.070 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:48.067921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:48.079 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:48.077207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:58.189 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:58.186837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:37:58.201 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:37:58.198999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:08.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:08.269188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:08.283 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:08.280901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:18.364 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:18.361756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:18.457 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:18.454267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:28.541 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:28.538546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:28.552 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:28.550282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:38.660 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:38.657029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:38.671 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:38.668865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:48.768 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:48.765645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:48.779 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:48.776838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:58.856 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:58.853445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:38:58.866 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:38:58.864225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:08.971 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:08.968225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:08.983 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:08.981232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:19.062 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:19.059646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:19.073 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:19.071367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:29.187 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:29.184752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:29.199 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:29.197016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:39.301 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:39.298789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:39.312 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:39.309899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:49.413 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:49.409744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:49.425 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:49.422532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:59.498 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:59.495797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:39:59.509 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:39:59.506665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:09.583 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:09.577842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:09.590 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:09.588262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:19.704 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:19.702068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:19.715 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:19.712725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:29.813 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:29.810751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:29.823 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:29.820983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:39.930 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:39.927191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:39.939 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:39.936435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:50.042 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:50.038979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:40:50.051 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:40:50.049286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:00.126 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:00.123393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:00.228 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:00.223579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:10.334 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:10.331593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:10.346 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:10.343324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:20.467 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:20.465147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:20.482 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:20.479474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:30.585 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:30.582271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:30.598 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:30.595762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:40.686 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:40.684170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:40.696 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:40.693098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:50.854 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:50.851005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:41:50.864 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:41:50.861646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:00.977 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:00.975022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:00.990 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:00.988040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:11.095 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:11.092158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:11.105 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:11.103221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:21.203 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:21.201132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:21.215 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:21.213400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:31.339 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:31.336297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:31.349 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:31.347528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:41.443 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:41.440891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:41.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:41.450021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:51.572 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:51.569355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:42:51.583 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:42:51.581088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:01.696 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:01.693577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:01.710 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:01.707563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:11.848 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:11.845596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:11.859 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:11.857311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:21.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:21.951854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:21.965 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:21.963023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:32.074 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:32.072008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:32.084 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:32.082280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:42.216 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:42.213441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:42.225 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:42.223121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:52.313 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:52.310792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:43:52.323 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:43:52.320984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:02.384 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:02.381582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:02.394 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:02.392463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:12.454 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:12.451408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:12.466 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:12.463812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:22.576 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:22.573745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:22.588 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:22.586455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:32.691 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:32.688423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:32.701 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:32.698994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:42.886 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:42.884064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:42.897 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:42.895008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:52.998 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:52.995460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:44:53.123 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:44:53.009906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:03.132 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:03.129494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:03.141 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:03.139774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:13.243 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:13.240955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:13.255 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:13.252891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:23.335 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:23.333065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:23.345 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:23.343253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:33.428 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:33.425694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:33.436 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:33.434634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:43.526 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:43.523753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:43.537 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:43.535216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:53.635 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:53.632743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:45:53.647 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:45:53.644778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:03.710 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:03.707979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:03.721 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:03.719243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:13.776 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:13.772193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:13.785 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:13.783164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:23.873 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:23.870975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:23.883 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:23.881643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:33.967 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:33.965478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:33.978 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:33.975893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:44.095 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:44.092511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:44.105 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:44.103521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:54.215 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:54.210853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:46:54.222 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:46:54.219809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:04.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:04.307989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:04.322 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:04.320029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:14.435 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:14.432236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:14.446 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:14.444217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:24.551 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:24.549154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:24.561 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:24.558699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:34.642 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:34.640328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:34.715 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:34.712757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:38.438 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:38.435951 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T13:47:39.058 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:39.055934 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:47:39.046+0000 7fe30c255380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:47:39.058 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T13:47:39.046+0000 7fe30c255380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:47:39.059 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T13:47:39.046+0000 7fe30c255380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:47:39.059 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:47:44.785 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:44.783651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:44.796 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:44.794475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:54.893 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:54.890564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:47:54.902 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:47:54.900294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:05.000 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:04.997984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:05.010 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:05.008529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:15.128 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:15.126524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:15.140 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:15.138564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:25.264 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:25.262574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:25.274 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:25.272822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:35.356 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:35.353767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:35.367 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:35.365131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:45.447 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:45.445485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:45.458 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:45.456885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:55.555 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:55.552625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:48:55.571 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:48:55.569555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:05.679 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:05.676642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:05.690 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:05.688900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:15.836 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:15.832830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:15.849 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:15.845713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:25.967 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:25.965212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:25.978 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:25.976315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:36.089 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:36.086751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:36.102 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:36.100467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:46.214 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:46.212051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:46.225 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:46.223265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:56.300 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:56.298320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:49:56.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:49:56.308383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:06.380 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:06.378474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:06.391 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:06.389090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:16.491 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:16.489264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:16.503 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:16.501340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:26.599 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:26.596742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:26.609 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:26.607352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:36.695 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:36.692682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:36.742 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:36.740634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:46.819 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:46.816684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:46.830 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:46.828473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:56.980 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:56.977704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:50:56.989 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:50:56.987239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:07.058 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:07.056020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:07.112 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:07.110540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:17.191 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:17.189136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:17.200 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:17.198455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:27.276 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:27.274194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:27.287 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:27.285170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:37.419 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:37.416989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:37.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:37.428955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:47.527 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:47.525493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:47.537 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:47.535907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:57.607 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:57.605566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:51:57.616 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:51:57.614334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:07.686 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:07.684395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:07.695 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:07.692744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:17.762 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:17.760377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:17.772 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:17.770772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:27.844 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:27.842474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:27.855 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:27.853589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:37.969 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:37.967090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:37.979 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:37.977498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:48.105 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:48.102989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:48.115 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:48.113871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:58.214 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:58.211161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:52:58.225 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:52:58.222954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:08.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:08.303711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:08.316 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:08.314427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:18.399 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:18.397138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:18.412 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:18.410226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:28.527 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:28.524539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:28.538 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:28.535647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:38.651 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:38.648962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:38.661 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:38.659277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:48.726 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:48.724165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:48.761 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:48.759924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:58.853 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:58.851037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:53:58.924 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:53:58.922906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:09.076 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:09.074319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:09.088 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:09.086077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:19.185 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:19.175920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:19.194 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:19.193167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:29.319 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:29.317580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:29.330 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:29.328849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:39.392 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:39.389808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:39.403 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:39.401677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:49.489 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:49.487573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:49.499 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:49.497854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:59.600 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:59.598572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:54:59.611 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:54:59.609543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:09.707 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:09.704957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:09.716 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:09.714742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:19.850 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:19.848294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:19.860 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:19.858125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:29.923 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:29.921534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:29.934 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:29.932523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:40.019 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:40.017897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:40.066 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:40.064336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:50.129 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:50.127624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:55:50.137 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:55:50.135502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:00.232 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:00.229956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:00.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:00.240921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:10.351 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:10.349228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:10.361 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:10.359836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:20.467 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:20.465393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:20.475 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:20.474015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:30.562 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:30.560077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:30.573 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:30.571573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:40.664 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:40.662062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:40.675 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:40.673237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:50.740 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:50.738187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:56:50.753 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:56:50.751610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:00.848 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:00.846655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:00.859 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:00.857293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:10.927 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:10.925287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:10.938 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:10.936296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:21.011 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:21.008862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:21.076 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:21.074306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:31.181 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:31.179036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:31.193 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:31.192191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:41.271 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:41.269561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:41.283 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:41.281382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:51.383 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:51.381408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:57:51.394 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:57:51.392593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:01.485 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:01.483849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:01.497 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:01.495377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:11.590 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:11.588661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:11.602 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:11.601190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:21.768 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:21.766418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:21.777 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:21.776124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:32.144 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:32.142556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:32.154 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:32.152624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:42.236 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:42.234680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:42.246 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:42.244703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:52.374 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:52.372470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:58:52.383 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:58:52.381564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:02.515 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:02.513561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:02.526 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:02.524412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:12.622 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:12.620723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:12.631 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:12.629902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:22.759 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:22.757158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:22.770 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:22.768675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:32.862 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:32.860277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:32.876 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:32.874705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:42.946 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:42.944052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:42.957 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:42.955214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:53.069 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:53.067591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T13:59:53.080 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 13:59:53.078356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:03.186 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:03.184076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:03.201 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:03.198545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:13.296 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:13.294303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:13.308 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:13.305924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:23.412 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:23.410216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:23.423 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:23.421500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:33.524 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:33.522319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:33.537 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:33.535170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:43.621 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:43.619043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:43.632 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:43.630433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:53.758 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:53.756081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:00:53.768 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:00:53.766655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:03.905 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:03.903814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:03.916 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:03.914752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:14.012 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:14.010827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:14.023 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:14.021228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:24.132 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:24.130531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:24.144 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:24.142475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:34.280 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:34.278594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:34.291 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:34.290255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:44.405 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:44.402818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:44.415 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:44.413664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:54.497 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:54.495665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:01:54.508 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:01:54.507081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:04.581 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:04.579433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:04.591 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:04.590069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:14.670 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:14.668583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:14.685 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:14.684208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:24.763 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:24.761946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:24.821 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:24.819182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:34.946 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:34.944231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:34.955 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:34.953631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:45.064 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:45.062117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:45.073 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:45.072293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:55.141 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:55.139723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:02:55.200 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:02:55.198394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:05.277 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:05.275857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:05.308 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:05.301246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:15.391 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:15.389769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:15.401 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:15.400057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:25.510 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:25.508077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:25.520 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:25.519027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:35.595 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:35.593960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:35.606 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:35.604893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:45.774 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:45.772417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:45.783 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:45.781450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:55.852 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:55.850855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:03:55.863 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:03:55.862014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:05.944 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:05.942333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:05.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:05.952423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:16.064 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:16.062025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:16.076 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:16.074313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:19.063 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:19.062108 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T14:04:19.711 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:19.709104 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:04:19.702+0000 7f18b1f5e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:04:19.711 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T14:04:19.702+0000 7f18b1f5e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:04:19.712 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T14:04:19.702+0000 7f18b1f5e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:04:19.712 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:04:26.144 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:26.142309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:26.156 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:26.154507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:36.299 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:36.297496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:36.308 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:36.307241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:46.548 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:46.546274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:46.559 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:46.557693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:56.694 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:56.692817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:04:56.705 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:04:56.704235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:06.833 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:06.831895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:06.844 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:06.842470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:16.934 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:16.932065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:16.945 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:16.942905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:27.016 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:27.014087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:27.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:27.022743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:37.095 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:37.093142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:37.105 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:37.103948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:47.218 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:47.216290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:47.229 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:47.227409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:57.332 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:57.330446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:05:57.344 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:05:57.342140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:07.445 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:07.443216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:07.455 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:07.454011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:17.554 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:17.552472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:17.565 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:17.562799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:27.627 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:27.625319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:27.638 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:27.636949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:37.707 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:37.705221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:37.718 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:37.716073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:47.812 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:47.810461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:47.824 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:47.821949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:57.921 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:57.918187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:06:57.932 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:06:57.930336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:08.103 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:08.101609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:08.115 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:08.113202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:18.237 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:18.234819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:18.248 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:18.246221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:28.330 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:28.328355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:28.342 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:28.340665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:38.405 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:38.402816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:38.417 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:38.415078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:48.496 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:48.494260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:48.506 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:48.504702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:58.580 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:58.578408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:07:58.590 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:07:58.588055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:08.706 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:08.703266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:08.716 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:08.714465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:18.789 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:18.787306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:18.801 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:18.799341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:28.926 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:28.923237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:28.935 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:28.932842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:39.046 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:39.041954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:39.056 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:39.052845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:49.147 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:49.144729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:49.160 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:49.157680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:59.234 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:59.232221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:08:59.246 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:08:59.243703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:09.325 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:09.322730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:09.335 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:09.333318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:19.454 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:19.451068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:19.466 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:19.463893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:29.602 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:29.599161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:29.613 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:29.610796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:39.737 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:39.734499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:39.764 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:39.762439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:49.890 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:49.887434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:09:49.901 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:09:49.899454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:00.009 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:00.003187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:00.018 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:00.015919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:10.136 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:10.133516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:10.147 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:10.145401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:20.240 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:20.238031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:20.253 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:20.250867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:30.370 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:30.367847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:30.379 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:30.377110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:40.453 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:40.449965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:40.464 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:40.462257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:50.553 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:50.550216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:10:50.565 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:10:50.563241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:00.673 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:00.669941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:00.682 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:00.680304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:10.755 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:10.752078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:10.765 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:10.762453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:21.028 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:21.025468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:21.039 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:21.037155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:31.175 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:31.171922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:31.182 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:31.180211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:41.294 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:41.292016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:41.306 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:41.303560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:51.495 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:51.492116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:11:51.506 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:11:51.503987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:01.708 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:01.705177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:01.719 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:01.717504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:11.794 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:11.791371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:11.805 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:11.802917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:21.878 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:21.875482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:21.947 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:21.944860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:32.028 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:32.025355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:32.040 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:32.037847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:42.163 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:42.161044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:42.176 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:42.174136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:52.266 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:52.263749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:12:52.276 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:12:52.273905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:02.391 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:02.389029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:02.403 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:02.401281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:12.504 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:12.501010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:12.515 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:12.512443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:22.623 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:22.620592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:22.636 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:22.634015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:32.789 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:32.786885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:32.802 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:32.799986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:42.929 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:42.925767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:42.939 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:42.936989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:53.004 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:53.001113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:13:53.015 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:13:53.012407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:03.079 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:03.077027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:03.091 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:03.089052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:13.203 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:13.200544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:13.213 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:13.210218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:23.277 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:23.274736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:23.287 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:23.284778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:33.376 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:33.373182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:33.387 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:33.384961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:43.478 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:43.475320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:43.488 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:43.486164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:53.586 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:53.583311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:14:53.597 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:14:53.594568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:03.689 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:03.686054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:03.702 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:03.698294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:13.802 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:13.799068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:13.812 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:13.809799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:23.933 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:23.930064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:23.942 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:23.940021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:34.011 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:34.008408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:34.022 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:34.019885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:44.090 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:44.086903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:44.101 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:44.098483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:54.165 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:54.162476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:15:54.177 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:15:54.174445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:04.327 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:04.324653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:04.343 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:04.339998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:14.436 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:14.433272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:14.448 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:14.445351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:24.558 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:24.555815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:24.567 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:24.564677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:34.662 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:34.658869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:34.672 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:34.669498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:44.746 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:44.743472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:44.760 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:44.757613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:54.829 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:54.826037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:16:54.891 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:16:54.888047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:04.959 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:04.956955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:05.028 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:05.025597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:15.080 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:15.077499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:15.087 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:15.085009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:25.217 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:25.214522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:25.227 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:25.224415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:35.389 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:35.386755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:35.403 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:35.400608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:45.474 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:45.471021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:45.485 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:45.481841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:55.545 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:55.541267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:17:55.593 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:17:55.590144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:05.707 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:05.703855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:05.717 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:05.714550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:15.811 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:15.806875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:15.817 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:15.814957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:25.885 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:25.881845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:25.923 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:25.920498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:35.993 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:35.990144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:36.006 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:36.003546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:46.115 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:46.112092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:46.128 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:46.125445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:56.227 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:56.224074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:18:56.239 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:18:56.236794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:06.327 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:06.324690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:06.339 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:06.336170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:16.408 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:16.404670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:16.420 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:16.417509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:26.498 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:26.495296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:26.508 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:26.505349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:36.597 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:36.594162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:36.609 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:36.605928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:46.725 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:46.722312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:46.735 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:46.732943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:56.840 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:56.836731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:19:56.852 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:19:56.848994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:06.914 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:06.911358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:06.927 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:06.924258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:17.034 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:17.030908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:17.052 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:17.048932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:27.162 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:27.159519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:27.174 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:27.171989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:37.303 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:37.300262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:37.315 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:37.312610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:47.437 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:47.433822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:47.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:47.449685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:57.533 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:57.530048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:57.542 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:57.538619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:20:59.721 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:20:59.717953 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T14:21:00.342 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:00.338013 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:21:00.329+0000 7f01d88d7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:21:00.342 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T14:21:00.329+0000 7f01d88d7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:21:00.343 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T14:21:00.329+0000 7f01d88d7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:21:00.343 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:21:07.644 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:07.641182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:07.656 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:07.653020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:17.789 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:17.785889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:17.799 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:17.795851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:27.916 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:27.912304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:27.923 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:27.919557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:38.038 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:38.034834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:38.048 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:38.045479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:48.151 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:48.147961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:48.162 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:48.159372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:58.255 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:58.251845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:21:58.315 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:21:58.312138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:08.436 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:08.432582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:08.445 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:08.442528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:18.578 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:18.575522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:18.591 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:18.588253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:28.664 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:28.661371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:28.676 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:28.672239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:38.777 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:38.773722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:38.788 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:38.784972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:48.865 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:48.861883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:48.876 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:48.872950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:58.950 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:58.947150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:22:58.961 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:22:58.957500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:09.030 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:09.027670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:09.079 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:09.075643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:19.189 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:19.185783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:19.200 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:19.196797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:29.338 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:29.335046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:29.350 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:29.347601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:39.484 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:39.480689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:39.495 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:39.492752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:49.563 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:49.559701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:49.575 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:49.572263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:59.665 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:59.661710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:23:59.677 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:23:59.673746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:09.809 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:09.805791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:09.819 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:09.816483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:20.021 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:20.018104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:20.032 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:20.029372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:30.205 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:30.201375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:30.217 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:30.213878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:40.333 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:40.330573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:40.342 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:40.339311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:50.433 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:50.428656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:24:50.441 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:24:50.438161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:00.551 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:00.548333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:00.561 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:00.557938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:10.644 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:10.640486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:10.653 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:10.650510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:20.721 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:20.717692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:20.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:20.728556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:30.864 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:30.861090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:30.879 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:30.876171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:40.994 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:40.990840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:41.005 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:41.002121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:51.103 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:51.100256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:25:51.117 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:25:51.113923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:01.174 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:01.169507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:01.183 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:01.180038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:11.274 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:11.271107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:11.326 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:11.323529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:21.457 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:21.453828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:22.214 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:21.465110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:31.586 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:31.583158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:31.597 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:31.594572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:41.705 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:41.702241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:41.717 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:41.713698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:51.813 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:51.809468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:26:51.824 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:26:51.821293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:01.897 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:01.894498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:01.957 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:01.954320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:12.047 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:12.044386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:12.057 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:12.054831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:22.112 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:22.108146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:22.155 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:22.150990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:32.298 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:32.295293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:32.307 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:32.303433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:42.410 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:42.406590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:42.418 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:42.415903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:52.513 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:52.509777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:27:52.526 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:27:52.522505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:02.629 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:02.626255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:02.639 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:02.636290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:12.699 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:12.695046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:12.750 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:12.747085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:22.856 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:22.852529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:22.873 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:22.864451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:32.982 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:32.978832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:32.994 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:32.990812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:43.061 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:43.058028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:43.074 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:43.069680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:53.159 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:53.156019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:28:53.172 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:28:53.169312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:03.273 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:03.269850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:03.283 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:03.280649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:13.368 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:13.365376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:13.377 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:13.374287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:23.535 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:23.532381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:23.547 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:23.544310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:33.605 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:33.602231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:33.617 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:33.614159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:43.687 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:43.683659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:43.698 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:43.695240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:53.805 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:53.800539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:29:53.816 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:29:53.811235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:03.875 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:03.872582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:03.886 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:03.883243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:13.952 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:13.948759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:13.963 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:13.959801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:24.030 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:24.027443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:24.099 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:24.095984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:34.175 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:34.171754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:34.185 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:34.182578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:44.282 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:44.278799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:44.292 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:44.289619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:54.443 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:54.439680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:30:54.454 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:30:54.451149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:04.576 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:04.572209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:04.587 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:04.583352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:14.796 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:14.793342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:14.809 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:14.806384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:24.996 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:24.992549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:25.006 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:25.002944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:35.175 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:35.171364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:35.190 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:35.186830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:45.270 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:45.266814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:45.474 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:45.471068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:55.542 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:55.538628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:31:55.554 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:31:55.550990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:05.624 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:05.621064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:05.636 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:05.632913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:15.709 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:15.705820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:15.721 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:15.717740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:25.810 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:25.806250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:25.820 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:25.817331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:35.932 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:35.928845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:35.944 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:35.941055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:46.039 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:46.036139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:46.052 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:46.048766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:56.153 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:56.149504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:32:56.164 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:32:56.161398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:06.279 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:06.275831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:06.290 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:06.287213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:16.373 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:16.370246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:16.381 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:16.378351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:26.501 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:26.498122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:26.513 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:26.510406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:36.579 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:36.576326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:36.588 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:36.584874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:46.720 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:46.716861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:46.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:46.728817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:56.808 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:56.805139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:33:56.818 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:33:56.815246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:06.927 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:06.924068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:06.941 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:06.938424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:17.035 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:17.031836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:17.047 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:17.043099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:27.162 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:27.158380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:27.173 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:27.169718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:37.288 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:37.284627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:37.299 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:37.296230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:47.379 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:47.375655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:47.389 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:47.386088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:57.510 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:57.507034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:34:57.522 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:34:57.519697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:07.634 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:07.630752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:07.644 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:07.641125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:17.781 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:17.777239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:17.796 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:17.793345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:27.928 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:27.924890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:27.939 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:27.936153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:38.035 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:38.031732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:38.048 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:38.043932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:48.136 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:48.127485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:48.143 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:48.139039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:58.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:58.238022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:35:58.252 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:35:58.248687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:08.358 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:08.354756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:08.369 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:08.366263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:18.435 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:18.431988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:18.446 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:18.443169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:28.563 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:28.560214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:28.573 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:28.569655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:38.694 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:38.690439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:38.705 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:38.702180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:48.773 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:48.770102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:48.784 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:48.781051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:58.855 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:58.851541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:36:58.865 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:36:58.861879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:08.935 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:08.931615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:08.992 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:08.989077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:19.055 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:19.052293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:19.066 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:19.063325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:29.178 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:29.174616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:29.193 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:29.189813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:39.269 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:39.265715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:39.280 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:39.276535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:40.350 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:40.346555 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T14:37:40.959 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:40.954142 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:37:40.944+0000 7f7dd3e38380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:37:40.960 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T14:37:40.944+0000 7f7dd3e38380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:37:40.960 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T14:37:40.944+0000 7f7dd3e38380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:37:40.960 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:37:49.377 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:49.373358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:49.388 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:49.385597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:59.490 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:59.486874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:37:59.501 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:37:59.498493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:09.572 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:09.568693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:09.582 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:09.578181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:19.680 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:19.676130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:19.693 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:19.689702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:29.785 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:29.782345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:29.797 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:29.793767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:39.866 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:39.862772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:39.926 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:39.923398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:50.031 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:50.027972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:38:50.042 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:38:50.039388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:00.152 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:00.149010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:00.164 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:00.161479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:10.265 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:10.261666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:10.277 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:10.274018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:20.352 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:20.348950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:20.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:20.359473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:30.432 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:30.429057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:30.444 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:30.440922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:40.515 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:40.511305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:40.526 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:40.522848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:50.600 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:50.595382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:39:50.611 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:39:50.607123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:00.778 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:00.774404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:00.789 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:00.785981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:10.890 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:10.886119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:10.900 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:10.897356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:20.973 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:20.969515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:21.024 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:21.020435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:31.091 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:31.087368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:31.151 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:31.148151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:41.221 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:41.217533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:41.231 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:41.228541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:51.294 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:51.290862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:40:51.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:40:51.301937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:01.382 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:01.378916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:01.451 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:01.448449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:11.522 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:11.518317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:11.534 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:11.531437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:21.640 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:21.636418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:21.649 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:21.646283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:31.761 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:31.757345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:31.771 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:31.768414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:41.867 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:41.863540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:41.878 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:41.875255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:51.944 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:51.940767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:41:51.955 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:41:51.951788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:02.058 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:02.054816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:02.069 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:02.065793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:12.137 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:12.133819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:12.148 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:12.144783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:22.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:22.237514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:22.253 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:22.250050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:32.376 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:32.372376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:32.389 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:32.385861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:42.468 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:42.464575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:42.481 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:42.477512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:52.556 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:52.553230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:42:52.571 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:42:52.567497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:02.676 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:02.672988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:02.688 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:02.685599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:12.761 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:12.757702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:12.772 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:12.769243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:22.880 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:22.877031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:22.888 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:22.885347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:32.964 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:32.960637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:32.976 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:32.971865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:43.098 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:43.094281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:43.110 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:43.106407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:53.331 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:53.328156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:43:53.341 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:43:53.338555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:03.417 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:03.413295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:03.427 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:03.424502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:13.590 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:13.587142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:13.601 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:13.598258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:23.701 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:23.696694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:23.710 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:23.706989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:33.861 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:33.856328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:33.872 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:33.869189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:44.006 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:44.003130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:44.017 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:44.014635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:54.129 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:54.125958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:44:54.140 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:44:54.137472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:04.270 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:04.266839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:04.280 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:04.276972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:14.393 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:14.389807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:14.405 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:14.401491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:24.504 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:24.501313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:24.515 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:24.512061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:34.621 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:34.617474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:34.635 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:34.631210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:44.764 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:44.760894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:44.775 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:44.771719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:54.881 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:54.877728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:45:54.986 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:45:54.982943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:05.111 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:05.108144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:05.141 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:05.137238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:15.264 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:15.260031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:15.276 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:15.272614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:25.432 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:25.428583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:25.446 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:25.442764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:35.531 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:35.522072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:35.536 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:35.533022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:45.610 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:45.606443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:45.621 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:45.617809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:55.840 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:55.836658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:46:55.851 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:46:55.848374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:05.947 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:05.942983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:05.957 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:05.953613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:16.023 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:16.019214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:16.034 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:16.030807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:26.096 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:26.093004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:26.107 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:26.103825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:36.213 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:36.210227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:36.224 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:36.221020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:46.314 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:46.310386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:46.329 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:46.325566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:56.440 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:56.436353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:47:56.450 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:47:56.446777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:06.536 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:06.532789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:06.548 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:06.544975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:16.642 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:16.638674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:16.653 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:16.649651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:26.731 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:26.726353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:26.743 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:26.739709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:36.808 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:36.803751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:36.816 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:36.813355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:46.910 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:46.905786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:46.921 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:46.917519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:56.987 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:56.983653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:48:57.000 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:48:56.996581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:07.102 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:07.098668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:07.113 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:07.109867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:17.176 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:17.172387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:17.187 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:17.184172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:27.284 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:27.279665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:27.292 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:27.289529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:37.394 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:37.390396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:37.407 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:37.404352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:47.472 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:47.468629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:47.482 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:47.478906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:57.593 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:57.589417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:49:57.602 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:49:57.599606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:07.669 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:07.666193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:07.680 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:07.676477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:17.794 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:17.790813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:17.805 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:17.802200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:27.918 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:27.914514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:27.932 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:27.927346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:38.002 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:37.998444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:38.059 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:38.055335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:48.154 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:48.150160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:48.168 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:48.164167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:58.280 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:58.276311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:50:58.290 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:50:58.286969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:08.358 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:08.354321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:08.369 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:08.366170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:18.485 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:18.481199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:18.512 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:18.504818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:28.640 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:28.636496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:28.650 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:28.647166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:38.732 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:38.728593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:38.743 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:38.739558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:48.808 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:48.804656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:48.819 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:48.815992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:58.920 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:58.916598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:51:58.931 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:51:58.928239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:08.993 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:08.986012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:09.002 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:08.998578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:19.079 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:19.075568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:19.128 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:19.125186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:29.222 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:29.218390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:29.233 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:29.229432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:39.313 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:39.309238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:39.323 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:39.320191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:49.400 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:49.396722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:49.413 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:49.409637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:59.544 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:59.540701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:52:59.555 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:52:59.552141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:09.665 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:09.661812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:09.674 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:09.671271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:19.752 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:19.748308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:19.761 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:19.757915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:29.873 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:29.870076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:29.884 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:29.880577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:40.005 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:40.001704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:40.017 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:40.013799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:50.121 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:50.118004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:53:50.133 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:53:50.129627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:00.241 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:00.237858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:00.253 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:00.249651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:10.322 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:10.319085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:10.333 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:10.329801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:20.438 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:20.434681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:20.450 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:20.446720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:20.967 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:20.963798 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T14:54:21.574 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:21.570161 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:54:21.560+0000 7ff81bd33380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:54:21.574 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T14:54:21.560+0000 7ff81bd33380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:54:21.574 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T14:54:21.560+0000 7ff81bd33380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:54:21.574 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:54:30.521 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:30.517721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:30.578 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:30.574608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:40.651 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:40.647452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:40.661 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:40.657782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:50.748 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:50.744698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:54:50.760 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:54:50.756907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:00.868 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:00.864484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:00.879 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:00.876187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:10.987 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:10.984094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:11.000 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:10.997163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:21.123 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:21.119153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:21.135 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:21.131721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:31.245 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:31.241369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:31.257 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:31.253839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:41.501 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:41.497978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:41.513 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:41.509750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:51.629 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:51.625156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:55:51.640 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:55:51.636900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:01.818 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:01.814512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:01.829 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:01.826342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:11.929 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:11.925999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:11.939 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:11.936185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:22.002 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:21.998598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:22.010 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:22.007488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:32.105 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:32.101258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:32.115 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:32.112479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:42.193 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:42.189596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:42.201 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:42.197229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:52.340 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:52.336713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:56:52.351 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:56:52.347953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:02.424 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:02.420957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:02.439 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:02.436011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:12.540 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:12.537158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:12.556 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:12.552735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:22.632 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:22.628746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:22.643 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:22.640283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:32.729 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:32.725896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:32.741 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:32.738108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:42.861 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:42.857713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:42.872 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:42.868947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:52.965 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:52.961900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:57:52.976 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:57:52.972621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:03.041 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:03.037403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:03.052 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:03.049088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:13.156 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:13.152731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:13.169 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:13.165691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:23.269 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:23.265472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:23.280 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:23.276596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:33.359 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:33.355315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:33.368 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:33.365137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:43.480 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:43.476865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:43.492 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:43.488826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:53.579 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:53.575870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:58:53.591 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:58:53.587854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:03.681 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:03.677428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:03.695 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:03.691542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:13.810 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:13.806988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:13.824 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:13.821411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:23.911 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:23.907022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:23.934 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:23.931204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:34.038 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:34.034115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:34.051 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:34.047602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:44.185 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:44.181630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:44.195 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:44.191612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:54.295 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:54.291468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T14:59:54.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 14:59:54.302067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:04.367 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:04.363525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:04.377 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:04.373812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:14.444 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:14.439998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:14.503 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:14.500274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:24.614 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:24.610570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:24.626 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:24.622080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:34.724 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:34.720483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:34.736 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:34.732473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:44.806 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:44.802601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:44.818 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:44.814640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:54.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:54.948012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:00:54.961 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:00:54.957528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:05.067 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:05.063888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:05.077 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:05.073890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:15.190 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:15.186450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:15.201 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:15.198226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:25.310 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:25.306755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:25.321 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:25.318081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:35.398 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:35.393920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:35.409 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:35.405858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:45.546 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:45.542021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:45.557 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:45.553141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:55.654 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:55.650729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:01:55.666 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:01:55.663000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:05.736 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:05.733025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:05.748 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:05.744633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:15.877 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:15.873721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:15.897 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:15.893621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:25.993 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:25.989664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:26.004 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:26.000522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:36.113 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:36.108686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:36.123 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:36.119483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:46.244 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:46.240083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:46.255 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:46.251605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:56.348 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:56.344405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:02:56.361 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:02:56.357213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:06.415 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:06.411155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:06.426 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:06.422503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:16.493 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:16.489930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:16.505 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:16.501349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:26.631 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:26.627303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:26.640 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:26.636786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:36.712 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:36.708902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:36.721 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:36.717756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:46.788 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:46.784411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:46.799 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:46.795835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:56.884 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:56.880900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:03:56.895 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:03:56.891982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:06.962 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:06.958596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:06.973 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:06.969406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:17.038 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:17.034097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:17.048 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:17.045126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:27.114 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:27.110540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:27.126 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:27.122305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:37.191 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:37.187333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:37.202 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:37.198891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:47.294 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:47.290755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:47.304 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:47.300892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:57.407 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:57.403689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:04:57.418 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:04:57.414569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:07.502 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:07.498695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:07.513 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:07.509717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:17.571 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:17.568148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:17.580 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:17.576847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:27.668 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:27.664040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:27.679 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:27.675460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:37.740 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:37.736624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:37.754 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:37.749942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:47.830 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:47.826407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:47.842 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:47.838639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:57.909 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:57.905810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:05:57.956 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:05:57.952909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:08.046 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:08.042780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:08.055 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:08.052040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:18.143 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:18.139104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:18.156 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:18.152800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:28.222 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:28.218787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:28.283 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:28.279290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:38.389 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:38.385075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:38.402 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:38.398603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:48.508 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:48.504197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:48.519 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:48.515952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:58.623 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:58.619038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:06:58.635 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:06:58.630112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:08.862 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:08.858455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:08.870 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:08.866375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:18.929 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:18.924855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:18.939 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:18.935326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:29.028 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:29.024015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:29.038 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:29.034947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:39.133 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:39.129272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:39.143 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:39.139895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:49.215 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:49.211536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:49.225 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:49.220881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:59.341 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:59.337211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:07:59.352 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:07:59.348914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:09.448 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:09.443073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:09.458 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:09.454396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:19.525 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:19.521523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:19.578 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:19.574618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:29.694 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:29.690178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:29.706 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:29.701354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:39.791 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:39.787306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:39.861 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:39.857081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:50.000 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:49.995583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:08:50.010 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:08:50.005349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:00.069 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:00.065988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:00.081 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:00.076499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:10.145 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:10.141678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:10.156 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:10.152700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:20.255 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:20.251740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:20.266 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:20.262841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:30.324 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:30.320632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:30.335 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:30.331347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:40.449 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:40.444547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:40.461 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:40.457280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:50.515 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:50.511358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:09:50.526 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:09:50.522989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:00.593 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:00.589053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:00.649 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:00.645751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:10.714 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:10.710750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:10.725 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:10.722119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:20.792 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:20.788202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:20.802 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:20.798456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:30.868 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:30.864179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:30.881 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:30.877398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:41.033 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:41.029656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:41.043 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:41.039720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:51.184 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:51.179776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:10:51.196 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:10:51.191901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:01.257 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:01.253227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:01.269 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:01.264408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:01.583 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:01.579146 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T15:11:02.226 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:02.222110 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T15:11:02.211+0000 7f47e6756380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T15:11:02.227 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T15:11:02.211+0000 7f47e6756380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T15:11:02.227 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T15:11:02.211+0000 7f47e6756380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T15:11:02.227 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T15:11:11.363 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:11.359150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:11.379 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:11.375372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:21.446 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:21.442244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:21.456 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:21.452431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:31.522 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:31.518070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:31.577 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:31.572990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:41.696 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:41.691790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:41.708 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:41.704196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:51.773 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:51.769432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:11:51.783 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:11:51.779402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:01.854 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:01.850098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:01.863 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:01.859445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:11.937 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:11.933061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:11.988 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:11.984075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:22.052 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:22.048192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:22.063 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:22.059444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:32.125 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:32.121219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:32.134 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:32.130886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:42.238 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:42.233935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:42.249 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:42.246022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:52.343 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:52.339609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:12:52.355 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:12:52.352006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:02.452 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:02.448189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:02.463 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:02.459383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:12.558 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:12.554182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:12.569 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:12.565416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:22.674 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:22.670557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:22.684 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:22.679975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:32.799 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:32.794154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:32.810 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:32.805619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:42.879 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:42.874811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:42.887 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:42.883959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:52.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:52.947493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:13:52.963 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:13:52.959801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:03.070 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:03.065707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:03.080 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:03.076947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:13.150 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:13.146283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:13.242 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:13.229812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:23.311 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:23.305705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:23.359 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:23.355019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:33.426 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:33.421957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:33.437 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:33.433733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:43.530 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:43.526494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:43.543 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:43.539355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:53.657 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:53.653040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:14:53.670 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:14:53.666608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:03.744 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:03.739975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:03.754 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:03.750562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:13.829 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:13.824729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:13.875 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:13.871346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:23.977 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:23.973784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:23.988 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:23.984791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:34.055 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:34.051431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:34.112 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:34.108321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:44.221 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:44.216939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:44.232 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:44.227993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:54.337 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:54.333640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:15:54.349 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:15:54.345566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:04.485 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:04.481818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:04.497 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:04.493638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:14.579 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:14.574794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:14.590 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:14.586995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:24.666 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:24.662636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:24.676 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:24.672543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:34.792 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:34.788303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:34.801 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:34.798191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:44.901 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:44.897464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:44.913 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:44.909683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:55.015 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:55.011894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:16:55.026 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:16:55.023128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:05.100 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:05.096384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:05.149 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:05.145314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:15.230 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:15.225988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:15.279 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:15.236293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:25.350 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:25.345764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:25.360 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:25.356430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:35.466 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:35.462453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:35.477 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:35.473025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:45.558 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:45.554461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:45.569 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:45.566115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:55.680 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:55.676796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:17:55.691 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:17:55.688080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:05.795 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:05.791307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:05.806 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:05.802364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:15.882 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:15.878269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:15.926 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:15.921975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:25.999 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:25.995200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:26.067 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:26.063986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:36.176 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:36.172533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:36.188 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:36.184614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:46.272 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:46.268331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:46.284 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:46.280749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:56.386 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:56.380468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:18:56.396 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:18:56.392721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:06.493 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:06.489823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:06.505 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:06.501446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:16.622 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:16.618969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:16.636 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:16.631964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:27.020 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:27.016154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:27.031 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:27.028130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:37.319 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:37.315580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:37.331 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:37.327731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:47.443 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:47.439481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:47.453 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:47.449677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:57.557 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:57.553480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:19:57.569 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:19:57.564961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:07.621 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:07.617525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:07.632 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:07.628291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:17.739 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:17.735396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:17.749 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:17.745745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:27.844 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:27.840323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:27.855 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:27.851526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:37.940 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:37.936391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:37.953 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:37.949903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:48.037 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:48.033480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:48.049 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:48.045346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:58.118 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:58.114804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:20:58.129 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:20:58.125151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:08.240 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:08.236389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:08.248 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:08.245185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:18.335 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:18.331472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:18.347 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:18.343665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:28.443 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:28.439008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:28.453 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:28.449706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:38.527 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:38.523431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:38.539 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:38.535478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:48.611 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:48.607614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:48.625 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:48.621470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:58.725 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:58.721908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:21:58.736 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:21:58.733264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:08.800 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:08.794979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:08.811 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:08.807057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:18.910 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:18.905859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:18.921 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:18.917970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:28.993 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:28.989359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:29.005 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:29.001220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:39.081 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:39.077387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:39.093 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:39.090177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:49.169 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:49.166050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:49.212 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:49.207888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:59.306 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:59.302606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:22:59.317 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:22:59.313753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:09.404 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:09.400731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:09.415 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:09.411391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:19.509 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:19.505521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:19.519 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:19.515724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:29.627 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:29.623181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:29.684 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:29.636455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:39.742 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:39.738185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:39.752 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:39.748964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:49.859 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:49.855208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:49.868 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:49.865316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:59.951 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:59.946588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:23:59.966 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:23:59.959232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:10.032 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:10.028167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:10.043 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:10.039973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:20.134 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:20.130128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:20.142 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:20.139294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:30.219 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:30.215415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:30.229 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:30.225129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:40.297 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:40.293308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:40.372 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:40.368632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:50.466 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:50.461909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:24:50.477 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:24:50.473333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:00.599 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:00.595426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:00.610 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:00.606950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:10.683 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:10.679354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:10.694 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:10.690671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:20.768 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:20.764775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:20.784 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:20.780455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:30.859 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:30.855570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:30.871 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:30.867871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:40.957 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:40.954013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:40.969 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:40.965603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:51.025 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:51.021769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:25:51.037 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:25:51.033700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:01.165 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:01.160941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:01.178 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:01.174694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:11.282 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:11.278953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:11.294 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:11.290414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:21.371 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:21.367106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:21.382 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:21.378588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:31.444 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:31.440404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:31.455 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:31.451558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:41.528 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:41.524897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:41.539 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:41.536230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:51.613 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:51.609093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:26:51.624 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:26:51.620609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:01.745 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:01.741013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:01.765 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:01.761131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:11.847 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:11.843547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:11.857 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:11.854249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:21.960 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:21.956052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:21.972 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:21.968712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:32.047 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:32.043443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:32.055 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:32.051973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:42.122 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:42.118096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:42.131 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:42.127497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:42.232 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:42.228812 I | op-mon: parsing mon endpoints: a=10.97.79.62:6789 2022-04-09T15:27:42.841 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:42.837823 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T15:27:42.830+0000 7f7c53bd3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T15:27:42.842 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T15:27:42.830+0000 7f7c53bd3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T15:27:42.842 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09T15:27:42.830+0000 7f7c53bd3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T15:27:42.842 INFO:tasks.rook.operator.smithi052.stdout:couldn't init storage provider. : exit status 5 2022-04-09T15:27:52.222 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:52.218218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:27:52.231 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:27:52.228036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:02.293 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:02.289950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:02.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:02.301856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:12.369 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:12.365182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:12.381 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:12.377760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:22.481 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:22.477545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:22.495 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:22.491060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:32.564 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:32.560895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:32.604 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:32.600610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:42.715 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:42.711911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:42.726 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:42.723213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:52.846 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:52.842683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:28:52.857 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:28:52.854026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:02.954 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:02.950226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:02.965 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:02.961920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:13.036 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:13.032854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:13.094 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:13.091251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:23.158 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:23.154455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:23.340 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:23.163357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:33.305 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:33.301505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:33.315 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:33.311919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:43.409 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:43.405838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:43.421 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:43.417381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:53.536 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:53.532232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:29:53.548 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:29:53.544556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:03.698 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:03.693729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:03.709 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:03.705445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:13.806 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:13.802310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:13.816 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:13.812603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:23.885 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:23.881689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:23.898 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:23.894196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:33.975 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:33.971734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:33.986 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:33.983280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:44.129 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:44.125331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:44.140 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:44.136973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:54.228 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:54.223994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:30:54.238 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:30:54.235160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:04.485 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:04.481191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:04.501 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:04.497434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:14.608 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:14.604066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:14.620 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:14.616845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:24.688 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:24.679150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:24.691 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:24.688101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:34.934 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:34.930125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:34.965 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:34.961438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:45.040 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:45.036299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:45.049 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:45.045796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:55.121 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:55.117096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:31:55.131 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:31:55.128028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:05.231 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:05.227521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:05.244 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:05.240662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:15.367 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:15.363518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:15.381 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:15.377920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:25.489 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:25.485187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:25.535 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:25.530843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:35.596 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:35.592809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:35.607 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:35.603546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:45.705 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:45.701690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:45.716 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:45.712710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:55.804 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:55.800197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:32:55.814 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:32:55.811078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:05.913 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:05.909616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:05.925 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:05.921502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:16.015 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:16.011567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:16.025 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:16.021252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:26.129 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:26.124953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:26.139 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:26.136078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:36.271 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:36.267264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:36.281 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:36.277825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:46.416 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:46.412351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:46.427 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:46.423532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:56.532 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:56.528634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:33:56.544 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:33:56.540652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:06.622 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:06.618309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:06.634 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:06.630773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:16.753 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:16.749802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:16.766 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:16.762799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:26.896 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:26.892005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:26.908 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:26.903787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:36.984 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:36.979335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:36.993 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:36.989410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:47.152 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:47.148738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:47.163 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:47.159813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:57.343 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:57.339220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:34:57.352 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:34:57.348554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:07.431 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:07.427201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:07.444 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:07.440250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:17.509 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:17.505407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:17.520 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:17.516309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:27.608 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:27.604321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:27.618 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:27.614866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:37.734 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:37.730360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:37.746 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:37.742716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:47.900 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:47.896321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:47.910 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:47.906430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:57.980 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:57.976387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:35:58.045 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:35:58.041424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:08.125 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:08.121221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:08.138 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:08.134113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:18.207 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:18.203682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:18.259 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:18.255087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:28.332 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:28.328796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:28.389 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:28.385486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:38.488 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:38.484664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:38.502 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:38.499120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:48.607 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:48.603715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:48.618 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:48.614730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:58.736 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:58.732164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:36:58.748 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:36:58.744951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:08.815 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:08.811035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:08.828 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:08.824561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:18.924 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:18.920130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:18.942 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:18.937987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:29.058 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:29.054510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:29.068 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:29.064224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:39.179 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:39.174483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:39.190 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:39.186815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:49.376 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:49.371983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:49.385 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:49.382035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:59.492 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:59.487643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:37:59.503 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:37:59.498746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:09.596 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:09.592090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:09.608 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:09.604641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:19.724 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:19.720209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:19.733 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:19.729845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:29.841 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:29.837200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:29.857 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:29.852212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:39.932 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:39.928296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:39.941 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:39.937353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:50.048 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:50.044360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:38:50.059 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:38:50.056124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:39:00.127 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:39:00.123725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:39:00.136 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:39:00.132969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:39:10.200 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:39:10.196019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T15:39:10.254 INFO:tasks.rook.operator.smithi052.stdout:2022-04-09 15:39:10.249653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-04-09T18:00:58.392 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-04-09T18:00:58.420 DEBUG:teuthology.task.console_log:Killing console logger for smithi052 2022-04-09T18:00:58.422 DEBUG:teuthology.exit:Finished running handlers