2022-03-10T01:14:17.826 INFO:root:teuthology version: 1.1.0-ff5db916 2022-03-10T01:14:17.828 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} --verbose --archive /home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728462 --name yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728462/orig.config.yaml 2022-03-10T01:14:17.864 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-10T01:14:17.927 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728462 branch: wip-yuri5-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} email: ceph-qa@ceph.io first_in_suite: false job_id: '6728462' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2022-03-07-0958 ceph: conf: global: osd crush chooseleaf type: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_branch: v1.7.2 rook_image: rook/ceph:v1.7.2 workunit: branch: wip-yuri5-testing-2022-03-07-0958 sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri5-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 targets: smithi186.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4z606pcG1UrpQFW6l4Ir0x5rfQ2Cclm4mF3dpULsnIapijN8X6qpZ5WqqsfGQJ0be2dgFKfSU5EJYqzM/g1uOR5t6XWFZevyeTDRzU4aFxiRk/UhVZUzSHMIpYyjekmyt/pFWs0hEeS7p0vL3tmgZTvHSfhr+6hdDt+l0MFXaryRTiRLtBH06y3m5RnH0erC/awxwuXjXJ7c/COZHh8R3jcqtm5rEciquGaGgAI8qtEucObl0iCHf0m1RXTqAfMo3G/BGtYhSOYt7RzH3WF+kSno9uQxEh2Gdwm8AqpTFzMudP0sD4JGBcEOl7gqeCtrtUhtMHwK3NTnWI3H09dq3zVhXRTZT4nMuLpb6Q+aV3LHHdU/PUvU8xN+3yfdjRfMfuj4SK4pELBfjiuLn5XFeIna6fWmIRzBShw9Y458ylTy7Oz3N2CwQmtBvmJcCI+rHlykmtlVUU9uFNaY6Lu9ZtEjtzsHklFbE/oUgWgoB7PJ3/kQP0N4Hj8Ise9oprkU= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - install: host.a: null - radosbench: clients: - client.a - rook.shell: commands: - 'ceph orch host label add `hostname` foo ceph orch host ls | grep foo ceph orch host label rm `hostname` foo ceph orch host ls | grep -v foo ' - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: ff5db9169566776b2f41faf671316ca58a318ea6 timestamp: 2022-03-10_01:04:51 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-10T01:14:17.928 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa; will attempt to use it 2022-03-10T01:14:17.929 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks 2022-03-10T01:14:17.929 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-10T01:14:17.931 INFO:teuthology.task.internal:Checking packages... 2022-03-10T01:14:17.953 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7a44e9ebca556862e9704be295a07719d7db47b3' 2022-03-10T01:14:17.953 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-10T01:14:17.954 INFO:teuthology.packaging:ref: None 2022-03-10T01:14:17.954 INFO:teuthology.packaging:tag: None 2022-03-10T01:14:17.955 INFO:teuthology.packaging:branch: wip-yuri5-testing-2022-03-07-0958 2022-03-10T01:14:17.955 INFO:teuthology.packaging:sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-10T01:14:17.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2022-03-07-0958 2022-03-10T01:14:18.107 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11025-g7a44e9eb-1focal 2022-03-10T01:14:18.107 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-10T01:14:18.117 INFO:teuthology.task.internal:no buildpackages task found 2022-03-10T01:14:18.118 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-10T01:14:18.218 INFO:teuthology.task.internal:Saving configuration 2022-03-10T01:14:18.231 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-10T01:14:18.318 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-10T01:14:18.340 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728462', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 01:06:52.141564', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBe7Mbq8GyJ46F0774d70hRMnn2Ir8n8WH1z0YeNy0HO'} 2022-03-10T01:14:18.340 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-10T01:14:18.401 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-10T01:14:18.401 INFO:teuthology.run_tasks:Running task console_log... 2022-03-10T01:14:18.545 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f190ecab9d8>, signals=[15]) 2022-03-10T01:14:18.546 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-10T01:14:18.642 INFO:teuthology.task.internal:Opening connections... 2022-03-10T01:14:18.643 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2022-03-10T01:14:18.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:14:18.698 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-10T01:14:18.709 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2022-03-10T01:14:18.825 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2022-03-10T01:14:18.825 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2022-03-10T01:14:18.832 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Ubuntu" 2022-03-10T01:14:18.832 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T01:14:18.833 INFO:teuthology.orchestra.run.smithi186.stdout:ID=ubuntu 2022-03-10T01:14:18.833 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE=debian 2022-03-10T01:14:18.833 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T01:14:18.834 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="20.04" 2022-03-10T01:14:18.834 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T01:14:18.834 INFO:teuthology.orchestra.run.smithi186.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T01:14:18.834 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T01:14:18.835 INFO:teuthology.orchestra.run.smithi186.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T01:14:18.835 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_CODENAME=focal 2022-03-10T01:14:18.835 INFO:teuthology.orchestra.run.smithi186.stdout:UBUNTU_CODENAME=focal 2022-03-10T01:14:18.836 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2022-03-10T01:14:18.853 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-10T01:14:18.867 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-10T01:14:18.900 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-10T01:14:18.900 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2022-03-10T01:14:18.907 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-10T01:14:18.981 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-10T01:14:18.982 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2022-03-10T01:14:18.991 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-10T01:14:19.156 INFO:teuthology.run_tasks:Running task kernel... 2022-03-10T01:14:19.216 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-10T01:14:19.217 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-10T01:14:19.217 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}} 2022-03-10T01:14:19.218 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-10T01:14:19.218 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2022-03-10T01:14:19.226 INFO:teuthology.orchestra.run.smithi186.stdout:5.4.0-97-generic 2022-03-10T01:14:19.226 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.4.0-97-generic 2022-03-10T01:14:19.227 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2022-03-10T01:14:19.295 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2022-03-10T01:14:19.578 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:14:19.578 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T01:14:19.633 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T01:14:19.981 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T01:14:20.054 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-10T01:14:20.204 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-10T01:14:20.364 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-10T01:14:20.542 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-10T01:14:20.559 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-10T01:14:20.560 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T01:14:20.561 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T01:14:20.601 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-10T01:14:20.602 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T01:14:20.605 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T01:14:20.606 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-10T01:14:20.621 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T01:14:20.636 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T01:14:20.637 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-10T01:14:20.637 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T01:14:20.638 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T01:14:20.658 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-10T01:14:20.663 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-10T01:14:20.710 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T01:14:20.720 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T01:14:20.720 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-10T01:14:20.739 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T01:14:20.739 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T01:14:20.740 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T01:14:20.740 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T01:14:20.751 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-10T01:14:20.764 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T01:14:20.770 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-10T01:14:20.811 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T01:14:20.812 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T01:14:20.813 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T01:14:20.813 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T01:14:20.814 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T01:14:20.816 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T01:14:20.906 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T01:14:20.907 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T01:14:20.907 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T01:14:20.908 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T01:14:20.908 INFO:teuthology.orchestra.run.smithi186.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T01:14:23.569 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 10.2 MB in 2s (4,601 kB/s) 2022-03-10T01:14:24.752 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:14:24.775 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T01:14:24.844 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:14:25.016 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-10T01:14:25.017 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-10T01:14:25.151 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2022-03-10T01:14:25.151 INFO:teuthology.orchestra.run.smithi186.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T01:14:25.151 INFO:teuthology.orchestra.run.smithi186.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T01:14:25.152 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T01:14:25.152 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2022-03-10T01:14:25.152 INFO:teuthology.orchestra.run.smithi186.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T01:14:25.153 INFO:teuthology.orchestra.run.smithi186.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T01:14:25.193 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2022-03-10T01:14:25.193 INFO:teuthology.orchestra.run.smithi186.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T01:14:25.193 INFO:teuthology.orchestra.run.smithi186.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T01:14:25.193 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T01:14:25.194 INFO:teuthology.orchestra.run.smithi186.stdout: thermald upower usbmuxd 2022-03-10T01:14:25.383 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 13 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T01:14:25.383 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 80.1 MB of archives. 2022-03-10T01:14:25.384 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T01:14:25.384 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T01:14:25.650 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T01:14:25.651 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T01:14:25.658 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T01:14:25.746 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T01:14:25.746 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T01:14:26.646 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T01:14:26.904 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T01:14:28.698 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T01:14:28.698 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T01:14:28.699 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T01:14:28.704 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T01:14:28.707 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T01:14:29.121 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 80.1 MB in 4s (22.8 MB/s) 2022-03-10T01:14:29.306 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T01:14:30.753 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T01:14:30.759 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T01:14:30.878 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T01:14:31.189 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T01:14:31.201 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T01:14:31.231 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T01:14:31.559 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T01:14:31.572 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T01:14:31.617 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T01:14:31.936 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T01:14:31.952 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T01:14:31.994 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T01:14:32.315 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T01:14:32.330 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T01:14:32.364 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T01:14:32.684 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T01:14:32.697 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:14:32.742 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:14:34.984 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T01:14:34.998 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:14:35.094 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:14:35.571 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T01:14:35.584 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:14:35.629 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:14:41.687 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T01:14:41.705 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T01:14:41.745 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T01:14:42.056 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T01:14:42.070 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T01:14:42.120 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T01:14:42.431 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package thermald. 2022-03-10T01:14:42.445 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T01:14:42.481 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T01:14:43.030 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package upower. 2022-03-10T01:14:43.045 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T01:14:43.098 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T01:14:43.495 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T01:14:43.509 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T01:14:43.553 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T01:14:43.854 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T01:14:43.976 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T01:14:44.102 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T01:14:44.228 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T01:14:44.354 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T01:14:44.480 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T01:14:44.598 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T01:14:45.007 INFO:teuthology.orchestra.run.smithi186.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T01:14:45.049 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T01:14:45.225 INFO:teuthology.orchestra.run.smithi186.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T01:14:45.239 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T01:14:45.239 INFO:teuthology.orchestra.run.smithi186.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T01:14:45.716 INFO:teuthology.orchestra.run.smithi186.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T01:14:45.937 INFO:teuthology.orchestra.run.smithi186.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T01:14:45.979 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T01:14:46.381 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T01:14:46.381 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T01:14:46.858 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:14:48.136 INFO:teuthology.orchestra.run.smithi186.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T01:14:48.136 INFO:teuthology.orchestra.run.smithi186.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T01:14:48.285 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:14:49.611 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:14:50.921 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T01:14:51.072 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T01:14:55.502 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T01:14:56.187 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T01:14:56.287 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:14:56.473 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T01:14:56.474 INFO:teuthology.orchestra.run.smithi186.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:15:19.722 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T01:15:19.722 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub' 2022-03-10T01:15:19.732 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T01:15:19.751 INFO:teuthology.orchestra.run.smithi186.stdout:Generating grub configuration file ... 2022-03-10T01:15:20.031 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T01:15:20.059 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:15:20.443 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T01:15:20.454 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T01:15:23.101 INFO:teuthology.orchestra.run.smithi186.stdout:done 2022-03-10T01:15:23.434 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T01:15:23.467 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T01:15:23.468 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2022-03-10T01:15:23.468 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2022-03-10T01:15:23.469 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2022-03-10T01:15:23.469 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 19 2022-03-10T01:15:23.469 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T01:15:23.470 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2022-03-10T01:15:23.473 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta-hwe-5.13 2022-03-10T01:15:23.473 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T01:15:23.473 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T01:15:23.474 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T01:15:23.474 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2022-03-10T01:15:23.474 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2022-03-10T01:15:23.474 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2022-03-10T01:15:23.475 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2022-03-10T01:15:23.475 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T01:15:23.475 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T01:15:23.475 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T01:15:23.476 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-10T01:15:23.476 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T01:15:23.476 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2022-03-10T01:15:23.539 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2022-03-10T01:15:23.690 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:15:23.806 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:15:23.897 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:15:23.980 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:15:25.215 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:15:25.244 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T01:15:25.311 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:15:25.483 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-10T01:15:25.484 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-10T01:15:25.657 INFO:teuthology.orchestra.run.smithi186.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T01:15:25.658 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T01:15:25.659 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T01:15:25.681 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T01:15:25.681 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2022-03-10T01:15:25.682 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2022-03-10T01:15:25.683 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2022-03-10T01:15:25.683 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 19 2022-03-10T01:15:25.683 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T01:15:25.683 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2022-03-10T01:15:25.684 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta-hwe-5.13 2022-03-10T01:15:25.684 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T01:15:25.684 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T01:15:25.684 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T01:15:25.684 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2022-03-10T01:15:25.685 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2022-03-10T01:15:25.686 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2022-03-10T01:15:25.686 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2022-03-10T01:15:25.686 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2022-03-10T01:15:25.693 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.wMwY08xBrn 2022-03-10T01:15:25.694 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub/grub.cfg /tmp/tmp.wMwY08xBrn 2022-03-10T01:15:25.750 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.wMwY08xBrn 2022-03-10T01:15:25.792 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.wMwY08xBrn is 8KB 2022-03-10T01:15:25.808 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.wMwY08xBrn 2022-03-10T01:15:25.815 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T01:15:25.868 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:15:25.869 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T01:15:25.869 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T01:15:25.902 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T01:15:25.902 DEBUG:teuthology.orchestra.run.smithi186:> sudo update-grub 2022-03-10T01:15:27.039 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub' 2022-03-10T01:15:27.041 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T01:15:27.048 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2022-03-10T01:15:27.285 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T01:15:27.297 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:15:27.699 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T01:15:27.711 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T01:15:29.268 INFO:teuthology.orchestra.run.smithi186.stderr:done 2022-03-10T01:15:29.271 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2022-03-10T01:15:59.305 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-10T01:15:59.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-03-10T01:15:59.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:16:17.747 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2022-03-10T01:16:47.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-03-10T01:16:47.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:17:19.793 DEBUG:teuthology.orchestra.run.smithi186:> true 2022-03-10T01:17:20.740 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-03-10T01:17:20.741 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-10T01:17:20.741 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2022-03-10T01:17:20.752 INFO:teuthology.orchestra.run.smithi186.stdout:5.13.0-35-generic 2022-03-10T01:17:20.752 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T01:17:20.752 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T01:17:20.753 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T01:17:21.754 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-10T01:17:21.755 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T01:17:21.839 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2022-03-10T01:17:21.883 DEBUG:teuthology.parallel:result is None 2022-03-10T01:17:21.883 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-10T01:17:21.901 INFO:teuthology.task.internal:Creating test directory... 2022-03-10T01:17:21.904 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T01:17:21.924 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-10T01:17:21.953 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-10T01:17:21.987 INFO:teuthology.task.internal:Creating archive directory... 2022-03-10T01:17:21.988 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T01:17:22.084 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-10T01:17:22.092 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-10T01:17:22.092 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T01:17:22.114 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T01:17:22.124 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T01:17:22.124 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-10T01:17:22.182 INFO:teuthology.task.internal:Configuring sudo... 2022-03-10T01:17:22.184 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T01:17:22.201 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-10T01:17:22.329 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-10T01:17:22.330 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T01:17:22.337 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T01:17:22.395 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T01:17:22.450 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:17:22.451 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T01:17:22.610 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2022-03-10T01:17:22.720 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-10T01:17:22.735 INFO:teuthology.task.internal:Starting timer... 2022-03-10T01:17:22.736 INFO:teuthology.run_tasks:Running task pcp... 2022-03-10T01:17:22.900 INFO:teuthology.run_tasks:Running task selinux... 2022-03-10T01:17:22.983 DEBUG:teuthology.task.selinux:Excluding smithi186: OS 'ubuntu' does not support SELinux 2022-03-10T01:17:22.984 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-10T01:17:22.984 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-10T01:17:22.985 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-10T01:17:22.985 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-10T01:17:23.078 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-10T01:17:23.079 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-10T01:17:23.122 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-10T01:17:23.125 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-10T01:21:06.226 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2022-03-10T01:21:06.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-03-10T01:21:06.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:21:06.510 DEBUG:teuthology.orchestra.run.smithi186:> true 2022-03-10T01:21:06.624 INFO:teuthology.run_tasks:Running task clock... 2022-03-10T01:21:06.730 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-10T01:21:06.731 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T01:21:06.734 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T01:21:06.812 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T01:21:06.813 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: Command line: ntpd -gq 2022-03-10T01:21:06.813 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: proto: precision = 0.100 usec (-23) 2022-03-10T01:21:06.813 INFO:teuthology.orchestra.run.smithi186.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T01:21:06.839 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T01:21:06.840 INFO:teuthology.orchestra.run.smithi186.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T01:21:06.843 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T01:21:06.844 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T01:21:06.844 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T01:21:06.844 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T01:21:06.844 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: Listen normally on 3 enp3s0f1 172.21.15.186:123 2022-03-10T01:21:06.845 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: Listen normally on 4 lo [::1]:123 2022-03-10T01:21:06.845 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6df5%5]:123 2022-03-10T01:21:06.845 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:06 ntpd[9019]: Listening on routing socket on fd #22 for interface updates 2022-03-10T01:21:10.836 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:10.837 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:10.837 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:10.837 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:10.837 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:10.838 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:10.838 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:10.838 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:10.859 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:10.859 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:10.859 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:10.860 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:10.860 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:10.860 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:10.860 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:10.860 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:10.913 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:10.931 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:10 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:11.806 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:11 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:11.806 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:11 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:11.815 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:11 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:11.815 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:11 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:11.815 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:11 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:11.815 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:11 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:12.827 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:12.828 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:12.828 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:12.828 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:12.828 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:12.829 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:12.829 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:12.829 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:12.829 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:12.830 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:12.830 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:12.830 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:12.831 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:12.833 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:12.833 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:12.834 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:12.836 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:12.837 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:12 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:13.807 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:13 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:13.808 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:13 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:13.808 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:13 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:13.808 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:13 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:13.808 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:13 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:13.809 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:13 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:14.810 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:14.810 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:14.864 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:14.864 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:14.864 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:14.864 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:14.879 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:14.880 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:14.880 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:14.880 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:14.881 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:14.881 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:14.881 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:14.881 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:14.882 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:14.882 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:14.882 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:14.927 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:14 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:15.805 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:15 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:15.806 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:15 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:15.806 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:15 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:15.807 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:15 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:15.807 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:15 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:15.807 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:15 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:16.817 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:16.818 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:16.818 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:16.837 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:16.837 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:16.838 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:16.838 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:21:16.838 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:21:16.838 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:21:16.839 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:21:16.839 INFO:teuthology.orchestra.run.smithi186.stdout:10 Mar 01:21:16 ntpd[9019]: ntpd: time slew -0.001785 s 2022-03-10T01:21:16.876 INFO:teuthology.orchestra.run.smithi186.stdout:ntpd: time slew -0.001785s 2022-03-10T01:21:16.877 INFO:teuthology.orchestra.run.smithi186.stderr:10 Mar 01:21:16 ntpd[9019]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T01:21:16.878 INFO:teuthology.orchestra.run.smithi186.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T01:21:16.899 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================== 2022-03-10T01:21:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:21:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:21:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:21:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:21:16.919 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-10T01:21:17.027 INFO:tasks.kubeadm:Kubeadm start 2022-03-10T01:21:17.028 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-10T01:21:17.029 DEBUG:teuthology.orchestra.run.smithi186:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T01:21:17.072 INFO:teuthology.orchestra.run.smithi186.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T01:21:17.077 INFO:teuthology.orchestra.run.smithi186.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T01:21:17.083 INFO:teuthology.orchestra.run.smithi186.stdout:net.ipv4.ip_forward = 1 2022-03-10T01:21:17.138 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:21:17.161 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T01:21:17.226 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:21:17.227 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/docker/daemon.json 2022-03-10T01:21:17.302 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl restart docker || true 2022-03-10T01:21:18.339 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-10T01:21:18.454 INFO:tasks.kubeadm:172.21.15.186 is in 172.21.0.0/20 at pos 4025 2022-03-10T01:21:18.670 INFO:tasks.kubeadm:Pod subnet: 10.253.200.0/21 2022-03-10T01:21:18.697 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-10T01:21:18.738 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T01:21:18.813 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:21:18.813 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:21:18.813 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:21:18.950 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:21:19.079 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:21:19.171 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:21:19.264 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:21:20.427 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:21:20.611 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-10T01:21:20.611 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-10T01:21:20.688 INFO:teuthology.orchestra.run.smithi186.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T01:21:20.703 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:21:20.703 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:21:20.703 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:21:20.755 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:21:20.910 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-10T01:21:20.910 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-10T01:21:21.077 INFO:teuthology.orchestra.run.smithi186.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T01:21:21.078 INFO:teuthology.orchestra.run.smithi186.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T01:21:21.078 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2022-03-10T01:21:21.079 INFO:teuthology.orchestra.run.smithi186.stdout: apt-transport-https 2022-03-10T01:21:21.290 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 1 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T01:21:21.290 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 4,680 B of archives. 2022-03-10T01:21:21.293 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T01:21:21.293 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T01:21:21.526 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T01:21:21.526 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T01:21:21.527 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: falling back to frontend: Readline 2022-03-10T01:21:21.543 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T01:21:21.543 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T01:21:21.547 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: falling back to frontend: Teletype 2022-03-10T01:21:21.548 INFO:teuthology.orchestra.run.smithi186.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T01:21:21.591 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 4,680 B in 0s (22.7 kB/s) 2022-03-10T01:21:21.679 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T01:21:21.739 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T01:21:21.740 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T01:21:21.770 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T01:21:22.078 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T01:21:23.452 INFO:teuthology.orchestra.run.smithi186.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T01:21:23.471 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:21:23.471 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:21:23.472 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:21:23.604 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:21:23.715 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:21:23.800 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:21:23.894 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:21:24.325 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T01:21:24.420 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T01:21:24.541 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 63.3 kB in 1s (68.0 kB/s) 2022-03-10T01:21:25.373 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:21:25.550 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-10T01:21:25.551 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-10T01:21:25.642 INFO:teuthology.orchestra.run.smithi186.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T01:21:25.658 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:21:25.659 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:21:25.659 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:21:25.718 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:21:25.887 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-10T01:21:25.887 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-10T01:21:26.030 INFO:teuthology.orchestra.run.smithi186.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T01:21:26.030 INFO:teuthology.orchestra.run.smithi186.stdout:bridge-utils set to manually installed. 2022-03-10T01:21:26.030 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2022-03-10T01:21:26.053 INFO:teuthology.orchestra.run.smithi186.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T01:21:26.086 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2022-03-10T01:21:26.087 INFO:teuthology.orchestra.run.smithi186.stdout: nftables 2022-03-10T01:21:26.089 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2022-03-10T01:21:26.089 INFO:teuthology.orchestra.run.smithi186.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T01:21:26.281 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 8 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T01:21:26.281 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 73.6 MB of archives. 2022-03-10T01:21:26.282 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T01:21:26.282 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T01:21:26.369 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T01:21:26.465 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T01:21:26.607 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T01:21:27.100 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T01:21:28.140 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T01:21:29.085 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T01:21:29.300 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T01:21:29.707 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T01:21:29.707 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T01:21:29.708 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: falling back to frontend: Readline 2022-03-10T01:21:29.713 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T01:21:29.713 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T01:21:29.714 INFO:teuthology.orchestra.run.smithi186.stderr:debconf: falling back to frontend: Teletype 2022-03-10T01:21:29.720 INFO:teuthology.orchestra.run.smithi186.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T01:21:29.769 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 73.6 MB in 3s (21.6 MB/s) 2022-03-10T01:21:29.839 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package conntrack. 2022-03-10T01:21:29.886 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T01:21:29.887 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T01:21:29.949 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T01:21:30.278 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package cri-tools. 2022-03-10T01:21:30.291 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T01:21:30.334 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T01:21:31.405 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ebtables. 2022-03-10T01:21:31.418 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T01:21:31.479 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T01:21:31.765 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T01:21:31.783 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T01:21:31.843 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T01:21:33.294 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package socat. 2022-03-10T01:21:33.307 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T01:21:33.344 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T01:21:33.684 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package kubelet. 2022-03-10T01:21:33.700 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T01:21:33.748 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T01:21:36.194 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package kubectl. 2022-03-10T01:21:36.205 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T01:21:36.248 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T01:21:37.500 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package kubeadm. 2022-03-10T01:21:37.519 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T01:21:37.551 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T01:21:38.687 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T01:21:38.821 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T01:21:38.937 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T01:21:39.841 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T01:21:39.987 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T01:21:40.134 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T01:21:40.260 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T01:21:40.482 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T01:21:40.993 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T01:21:41.540 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T01:21:42.400 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T01:21:57.139 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T01:22:00.641 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T01:22:03.263 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T01:22:07.537 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T01:22:09.270 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T01:22:17.960 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T01:22:20.552 INFO:teuthology.orchestra.run.smithi186.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T01:22:20.556 INFO:tasks.kubeadm:Token: abcdef.1uw5hrq3tnlamkn1 2022-03-10T01:22:20.557 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): '172.21.15.186'} 2022-03-10T01:22:20.557 DEBUG:teuthology.orchestra.run.smithi186:> sudo kubeadm init --node-name smithi186 --token abcdef.1uw5hrq3tnlamkn1 --pod-network-cidr 10.253.200.0/21 2022-03-10T01:22:20.875 INFO:teuthology.orchestra.run.smithi186.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-10T01:22:20.876 INFO:teuthology.orchestra.run.smithi186.stdout:[preflight] Running pre-flight checks 2022-03-10T01:22:21.268 INFO:teuthology.orchestra.run.smithi186.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-10T01:22:21.269 INFO:teuthology.orchestra.run.smithi186.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-10T01:22:21.269 INFO:teuthology.orchestra.run.smithi186.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-10T01:22:21.576 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-10T01:22:21.865 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "ca" certificate and key 2022-03-10T01:22:22.050 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "apiserver" certificate and key 2022-03-10T01:22:22.051 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi186] and IPs [10.96.0.1 172.21.15.186] 2022-03-10T01:22:22.488 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-10T01:22:22.565 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-10T01:22:22.800 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-10T01:22:22.892 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-10T01:22:23.035 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-10T01:22:23.036 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi186] and IPs [172.21.15.186 127.0.0.1 ::1] 2022-03-10T01:22:23.184 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-10T01:22:23.184 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi186] and IPs [172.21.15.186 127.0.0.1 ::1] 2022-03-10T01:22:23.306 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-10T01:22:23.558 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-10T01:22:23.644 INFO:teuthology.orchestra.run.smithi186.stdout:[certs] Generating "sa" key and public key 2022-03-10T01:22:23.645 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-10T01:22:23.709 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-10T01:22:24.155 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-10T01:22:24.262 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-10T01:22:24.460 INFO:teuthology.orchestra.run.smithi186.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-10T01:22:24.474 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T01:22:24.475 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T01:22:24.475 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet-start] Starting the kubelet 2022-03-10T01:22:24.679 INFO:teuthology.orchestra.run.smithi186.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-10T01:22:24.680 INFO:teuthology.orchestra.run.smithi186.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-10T01:22:24.684 INFO:teuthology.orchestra.run.smithi186.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-10T01:22:24.686 INFO:teuthology.orchestra.run.smithi186.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-10T01:22:24.687 INFO:teuthology.orchestra.run.smithi186.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-10T01:22:24.689 INFO:teuthology.orchestra.run.smithi186.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-10T01:22:41.695 INFO:teuthology.orchestra.run.smithi186.stdout:[apiclient] All control plane components are healthy after 17.003851 seconds 2022-03-10T01:22:41.696 INFO:teuthology.orchestra.run.smithi186.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-10T01:22:41.724 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-10T01:22:41.725 INFO:teuthology.orchestra.run.smithi186.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-10T01:22:42.357 INFO:teuthology.orchestra.run.smithi186.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-10T01:22:42.358 INFO:teuthology.orchestra.run.smithi186.stdout:[mark-control-plane] Marking the node smithi186 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-10T01:22:42.358 INFO:teuthology.orchestra.run.smithi186.stdout:[mark-control-plane] Marking the node smithi186 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-10T01:22:42.937 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] Using token: abcdef.1uw5hrq3tnlamkn1 2022-03-10T01:22:42.937 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-10T01:22:42.974 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-10T01:22:43.005 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-10T01:22:43.015 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-10T01:22:43.023 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-10T01:22:43.082 INFO:teuthology.orchestra.run.smithi186.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-10T01:22:43.123 INFO:teuthology.orchestra.run.smithi186.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-10T01:22:43.642 INFO:teuthology.orchestra.run.smithi186.stdout:[addons] Applied essential addon: CoreDNS 2022-03-10T01:22:43.850 INFO:teuthology.orchestra.run.smithi186.stdout:[addons] Applied essential addon: kube-proxy 2022-03-10T01:22:43.852 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:22:43.852 INFO:teuthology.orchestra.run.smithi186.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-10T01:22:43.853 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:22:43.853 INFO:teuthology.orchestra.run.smithi186.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-10T01:22:43.853 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:22:43.853 INFO:teuthology.orchestra.run.smithi186.stdout: mkdir -p $HOME/.kube 2022-03-10T01:22:43.853 INFO:teuthology.orchestra.run.smithi186.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-10T01:22:43.854 INFO:teuthology.orchestra.run.smithi186.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-10T01:22:43.854 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:22:43.854 INFO:teuthology.orchestra.run.smithi186.stdout:Alternatively, if you are the root user, you can run: 2022-03-10T01:22:43.854 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:22:43.855 INFO:teuthology.orchestra.run.smithi186.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-10T01:22:43.855 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:22:43.855 INFO:teuthology.orchestra.run.smithi186.stdout:You should now deploy a pod network to the cluster. 2022-03-10T01:22:43.855 INFO:teuthology.orchestra.run.smithi186.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-10T01:22:43.856 INFO:teuthology.orchestra.run.smithi186.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-10T01:22:43.856 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:22:43.856 INFO:teuthology.orchestra.run.smithi186.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-10T01:22:43.856 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:22:43.856 INFO:teuthology.orchestra.run.smithi186.stdout:kubeadm join 172.21.15.186:6443 --token abcdef.1uw5hrq3tnlamkn1 \ 2022-03-10T01:22:43.857 INFO:teuthology.orchestra.run.smithi186.stdout: --discovery-token-ca-cert-hash sha256:419d01fbec8345d9e86ce3ce7aa8d2ce9a29d5567ecb5d23772d8201cfd3717c 2022-03-10T01:22:43.858 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:22:43.859 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-10T01:22:43.872 INFO:tasks.kubeadm:Setting up kubectl 2022-03-10T01:22:43.873 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T01:22:43.888 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:22:43.889 DEBUG:teuthology.orchestra.run.smithi186:> dd of=.kube/config 2022-03-10T01:22:43.894 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:22:43.894 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/root/.kube/config 2022-03-10T01:22:43.950 DEBUG:teuthology.orchestra.run.smithi186:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-10T01:22:47.004 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-10T01:22:47.137 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-10T01:22:47.149 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-10T01:22:47.183 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-10T01:22:47.212 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-10T01:22:47.303 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-10T01:22:47.343 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-10T01:22:47.464 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-10T01:22:47.488 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-10T01:22:47.522 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-10T01:22:47.555 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-10T01:22:47.619 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-10T01:22:47.665 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-10T01:22:47.765 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-10T01:22:47.790 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-10T01:22:47.848 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-10T01:22:47.948 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-10T01:22:47.973 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-10T01:22:48.040 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-10T01:22:48.172 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-10T01:22:48.197 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-10T01:22:48.343 INFO:teuthology.orchestra.run.smithi186.stdout:namespace/tigera-operator created 2022-03-10T01:22:48.355 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-10T01:22:48.381 INFO:teuthology.orchestra.run.smithi186.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-10T01:22:48.392 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/tigera-operator created 2022-03-10T01:22:48.431 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-10T01:22:48.500 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-10T01:22:48.510 INFO:teuthology.orchestra.run.smithi186.stdout:deployment.apps/tigera-operator created 2022-03-10T01:22:48.518 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f - 2022-03-10T01:22:51.436 INFO:teuthology.orchestra.run.smithi186.stdout:installation.operator.tigera.io/default created 2022-03-10T01:22:51.440 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-10T01:22:51.441 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:22:51.441 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:22:51.447 INFO:tasks.kubeadm:Scratch devices on smithi186: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T01:22:51.448 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T01:22:51.476 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records in 2022-03-10T01:22:51.476 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2022-03-10T01:22:51.477 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0185254 s, 566 MB/s 2022-03-10T01:22:51.478 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T01:22:51.504 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records in 2022-03-10T01:22:51.505 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2022-03-10T01:22:51.505 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0146433 s, 716 MB/s 2022-03-10T01:22:51.507 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T01:22:51.586 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records in 2022-03-10T01:22:51.586 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2022-03-10T01:22:51.587 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0168884 s, 621 MB/s 2022-03-10T01:22:51.588 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T01:22:51.615 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records in 2022-03-10T01:22:51.615 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2022-03-10T01:22:51.616 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0165403 s, 634 MB/s 2022-03-10T01:22:51.624 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-10T01:22:51.624 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi186-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi186 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi186-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi186 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi186-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi186 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi186-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi186 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-10T01:22:51.625 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f - 2022-03-10T01:22:52.481 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-1 created 2022-03-10T01:22:52.530 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-2 created 2022-03-10T01:22:52.535 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-3 created 2022-03-10T01:22:52.551 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-4 created 2022-03-10T01:22:52.576 INFO:teuthology.orchestra.run.smithi186.stdout:storageclass.storage.k8s.io/scratch created 2022-03-10T01:22:52.580 DEBUG:teuthology.orchestra.run.smithi186:> kubectl taint node smithi186 node-role.kubernetes.io/master- || true 2022-03-10T01:22:52.685 INFO:teuthology.orchestra.run.smithi186.stdout:node/smithi186 untainted 2022-03-10T01:22:52.688 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-10T01:22:52.689 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-10T01:22:52.704 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-10T01:22:52.704 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:22:52.705 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:22:52.713 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T01:22:52.713 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T01:22:52.762 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T01:22:52.763 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:22:52.763 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-10T01:22:52.763 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:22:52.764 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-03-10 01:22:51.493821000 +0000 2022-03-10T01:22:52.764 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-03-10 01:22:51.489821000 +0000 2022-03-10T01:22:52.764 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-03-10 01:22:51.489821000 +0000 2022-03-10T01:22:52.764 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-03-10T01:22:52.771 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-03-10T01:22:52.771 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-03-10T01:22:52.772 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000606242 s, 845 kB/s 2022-03-10T01:22:52.774 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T01:22:52.779 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T01:22:52.780 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:22:52.780 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-10T01:22:52.781 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:22:52.781 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-03-10 01:22:51.509820999 +0000 2022-03-10T01:22:52.782 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-03-10 01:22:51.509820999 +0000 2022-03-10T01:22:52.782 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-03-10 01:22:51.509820999 +0000 2022-03-10T01:22:52.782 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-03-10T01:22:52.788 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-03-10T01:22:52.789 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-03-10T01:22:52.789 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000556726 s, 920 kB/s 2022-03-10T01:22:52.791 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T01:22:52.798 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T01:22:52.798 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:22:52.799 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-10T01:22:52.799 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:22:52.800 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-03-10 01:22:51.589820997 +0000 2022-03-10T01:22:52.800 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-03-10 01:22:51.589820997 +0000 2022-03-10T01:22:52.800 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-03-10 01:22:51.589820997 +0000 2022-03-10T01:22:52.800 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-03-10T01:22:52.803 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-03-10T01:22:52.804 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-03-10T01:22:52.804 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000489582 s, 1.0 MB/s 2022-03-10T01:22:52.808 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T01:22:52.857 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T01:22:52.857 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:22:52.858 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-10T01:22:52.858 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:22:52.858 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-03-10 01:22:51.621820997 +0000 2022-03-10T01:22:52.858 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-03-10 01:22:51.621820997 +0000 2022-03-10T01:22:52.859 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-03-10 01:22:51.621820997 +0000 2022-03-10T01:22:52.859 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-03-10T01:22:52.863 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-03-10T01:22:52.864 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-03-10T01:22:52.864 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000652949 s, 784 kB/s 2022-03-10T01:22:52.868 DEBUG:teuthology.orchestra.run.smithi186:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T01:22:53.016 INFO:teuthology.orchestra.run.smithi186.stdout:loop 2022-03-10T01:22:53.017 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_1... 2022-03-10T01:22:53.018 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T01:22:53.036 INFO:teuthology.orchestra.run.smithi186.stdout:1 2022-03-10T01:22:53.054 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_11 2022-03-10T01:22:53.072 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_2... 2022-03-10T01:22:53.072 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T01:22:53.090 INFO:teuthology.orchestra.run.smithi186.stdout:1 2022-03-10T01:22:53.111 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_21 2022-03-10T01:22:53.129 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_3... 2022-03-10T01:22:53.129 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T01:22:53.147 INFO:teuthology.orchestra.run.smithi186.stdout:1 2022-03-10T01:22:53.165 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_31 2022-03-10T01:22:53.182 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_4... 2022-03-10T01:22:53.183 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T01:22:53.247 INFO:teuthology.orchestra.run.smithi186.stdout:1 2022-03-10T01:22:53.268 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_41 2022-03-10T01:22:53.293 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:22:53.293 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:22:53.299 DEBUG:teuthology.orchestra.run.smithi186:> sudo nvme list 2022-03-10T01:22:53.308 INFO:teuthology.orchestra.run.smithi186.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T01:22:53.309 INFO:teuthology.orchestra.run.smithi186.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T01:22:53.309 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme0n1 CVFT623300CB400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-10T01:22:53.309 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme1n1 3e4ffd03d01b7d4a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:22:53.309 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme2n1 f7b7d14e6acea100 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:22:53.309 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme3n1 6c0959e48a4fcf60 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:22:53.311 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-10T01:22:54.312 DEBUG:teuthology.orchestra.run.smithi186:> sudo nvme list 2022-03-10T01:22:54.325 INFO:teuthology.orchestra.run.smithi186.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T01:22:54.326 INFO:teuthology.orchestra.run.smithi186.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T01:22:54.326 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme0n1 CVFT623300CB400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-10T01:22:54.326 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme1n1 3e4ffd03d01b7d4a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:22:54.326 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme2n1 f7b7d14e6acea100 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:22:54.327 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme3n1 6c0959e48a4fcf60 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:22:54.327 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme4n1 c7861b2629464d16 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:22:54.328 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T01:22:54.328 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:22:54.329 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/scratch_devs 2022-03-10T01:22:54.342 INFO:teuthology.run_tasks:Running task rook... 2022-03-10T01:22:54.419 INFO:tasks.rook:Rook start 2022-03-10T01:22:54.420 INFO:tasks.rook:Config: {'conf': {'global': {'osd crush chooseleaf type': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3', 'rook_branch': 'v1.7.2', 'rook_image': 'rook/ceph:v1.7.2'} 2022-03-10T01:22:54.420 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-10T01:22:54.420 INFO:tasks.rook:Cloning https://github.com/rook/rook branch v1.7.2 2022-03-10T01:22:54.421 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf rook && git clone --single-branch --branch v1.7.2 https://github.com/rook/rook rook 2022-03-10T01:22:54.430 INFO:teuthology.orchestra.run.smithi186.stderr:Cloning into 'rook'... 2022-03-10T01:23:07.470 INFO:teuthology.orchestra.run.smithi186.stderr:Note: switching to '1784489964d2e417b6050c7edc8f626b5cca49f5'. 2022-03-10T01:23:07.471 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:23:07.471 INFO:teuthology.orchestra.run.smithi186.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-10T01:23:07.471 INFO:teuthology.orchestra.run.smithi186.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-10T01:23:07.472 INFO:teuthology.orchestra.run.smithi186.stderr:state without impacting any branches by switching back to a branch. 2022-03-10T01:23:07.472 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:23:07.473 INFO:teuthology.orchestra.run.smithi186.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-10T01:23:07.473 INFO:teuthology.orchestra.run.smithi186.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-10T01:23:07.474 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:23:07.474 INFO:teuthology.orchestra.run.smithi186.stderr: git switch -c 2022-03-10T01:23:07.475 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:23:07.475 INFO:teuthology.orchestra.run.smithi186.stderr:Or undo this operation with: 2022-03-10T01:23:07.475 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:23:07.475 INFO:teuthology.orchestra.run.smithi186.stderr: git switch - 2022-03-10T01:23:07.476 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:23:07.476 INFO:teuthology.orchestra.run.smithi186.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-10T01:23:07.476 INFO:teuthology.orchestra.run.smithi186.stderr: 2022-03-10T01:23:07.565 INFO:tasks.rook:/home/teuthworker 2022-03-10T01:23:07.566 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/sbin:/usr/sbin ip addr show 2022-03-10T01:23:07.615 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-10T01:23:07.615 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-10T01:23:07.615 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2022-03-10T01:23:07.615 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2022-03-10T01:23:07.616 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2022-03-10T01:23:07.616 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2022-03-10T01:23:07.616 INFO:teuthology.orchestra.run.smithi186.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T01:23:07.616 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2c brd ff:ff:ff:ff:ff:ff 2022-03-10T01:23:07.617 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f0 2022-03-10T01:23:07.617 INFO:teuthology.orchestra.run.smithi186.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T01:23:07.617 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2d brd ff:ff:ff:ff:ff:ff 2022-03-10T01:23:07.617 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f1 2022-03-10T01:23:07.618 INFO:teuthology.orchestra.run.smithi186.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T01:23:07.618 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f4 brd ff:ff:ff:ff:ff:ff 2022-03-10T01:23:07.618 INFO:teuthology.orchestra.run.smithi186.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-10T01:23:07.618 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f5 brd ff:ff:ff:ff:ff:ff 2022-03-10T01:23:07.619 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.21.15.186/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-10T01:23:07.619 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft 42851sec preferred_lft 42851sec 2022-03-10T01:23:07.619 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::ec4:7aff:fe88:6df5/64 scope link 2022-03-10T01:23:07.619 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2022-03-10T01:23:07.620 INFO:teuthology.orchestra.run.smithi186.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-10T01:23:07.620 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 02:42:2f:40:e5:39 brd ff:ff:ff:ff:ff:ff 2022-03-10T01:23:07.620 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-10T01:23:07.620 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2022-03-10T01:23:07.621 DEBUG:teuthology.orchestra.run.smithi186:> which systemctl 2022-03-10T01:23:07.626 INFO:teuthology.orchestra.run.smithi186.stdout:/usr/bin/systemctl 2022-03-10T01:23:07.626 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-10T01:23:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:23:07.632 DEBUG:teuthology.orchestra.run.smithi186:> true 2022-03-10T01:23:07.709 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-03-10T01:23:07.759 INFO:teuthology.orchestra.run.smithi186.stdout:Ubuntu 2022-03-10T01:23:07.765 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-10T01:23:07.765 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-10T01:23:07.784 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-10T01:23:07.836 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:23:07.836 DEBUG:teuthology.orchestra.run.smithi186:> dd if=rook/cluster/examples/kubernetes/ceph/operator.yaml of=/dev/stdout 2022-03-10T01:23:07.846 INFO:tasks.rook:Patching operator to use image rook/ceph:v1.7.2 2022-03-10T01:23:07.895 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:23:07.896 DEBUG:teuthology.orchestra.run.smithi186:> dd of=operator.yaml 2022-03-10T01:23:07.903 INFO:tasks.rook:Deploying operator 2022-03-10T01:23:07.971 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-10T01:23:08.006 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-10T01:23:08.019 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml 2022-03-10T01:23:08.299 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-10T01:23:08.321 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-10T01:23:08.440 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-10T01:23:08.460 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-10T01:23:08.518 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-10T01:23:08.596 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-10T01:23:08.600 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-10T01:23:08.656 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-10T01:23:08.686 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-10T01:23:08.776 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-10T01:23:08.824 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-10T01:23:08.873 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-10T01:23:08.969 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-10T01:23:08.994 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-10T01:23:09.054 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created 2022-03-10T01:23:09.153 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created 2022-03-10T01:23:09.170 INFO:teuthology.orchestra.run.smithi186.stdout:customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created 2022-03-10T01:23:09.220 INFO:teuthology.orchestra.run.smithi186.stdout:namespace/rook-ceph created 2022-03-10T01:23:09.246 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-10T01:23:09.311 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-admission-controller created 2022-03-10T01:23:09.320 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role created 2022-03-10T01:23:09.347 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding created 2022-03-10T01:23:09.370 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-10T01:23:09.395 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T01:23:09.402 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T01:23:09.411 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-10T01:23:09.485 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-10T01:23:09.488 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-10T01:23:09.494 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-system created 2022-03-10T01:23:09.502 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T01:23:09.529 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T01:23:09.553 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-10T01:23:09.578 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-osd created 2022-03-10T01:23:09.646 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-mgr created 2022-03-10T01:23:09.649 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-10T01:23:09.670 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T01:23:09.721 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T01:23:09.838 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-10T01:23:09.842 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-10T01:23:09.859 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-10T01:23:09.888 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-10T01:23:09.922 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T01:23:09.994 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-10T01:23:09.997 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-10T01:23:10.011 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-10T01:23:10.022 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T01:23:10.032 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-10T01:23:10.044 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-10T01:23:10.045 INFO:teuthology.orchestra.run.smithi186.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-10T01:23:10.149 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-10T01:23:10.153 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-10T01:23:10.172 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-10T01:23:10.195 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-10T01:23:10.207 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-10T01:23:10.211 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-10T01:23:10.248 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-10T01:23:10.317 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-10T01:23:10.398 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-10T01:23:10.676 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-10T01:23:10.684 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-10T01:23:10.728 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-10T01:23:11.000 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-10T01:23:11.003 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-10T01:23:11.411 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-10T01:23:11.413 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-10T01:23:11.857 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-10T01:23:11.946 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-10T01:23:12.245 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-10T01:23:12.262 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-10T01:23:12.337 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T01:23:12.430 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-10T01:23:12.433 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-10T01:23:12.689 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-10T01:23:12.692 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T01:23:12.719 INFO:teuthology.orchestra.run.smithi186.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-10T01:23:12.727 INFO:teuthology.orchestra.run.smithi186.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-10T01:23:12.747 INFO:teuthology.orchestra.run.smithi186.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-10T01:23:12.852 INFO:teuthology.orchestra.run.smithi186.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-10T01:23:12.856 INFO:teuthology.orchestra.run.smithi186.stdout:configmap/rook-ceph-operator-config created 2022-03-10T01:23:12.903 INFO:teuthology.orchestra.run.smithi186.stdout:deployment.apps/rook-ceph-operator created 2022-03-10T01:23:12.909 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:23:13.022 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:23:13.023 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-p8gb8 0/1 Pending 0 1s 2022-03-10T01:23:23.026 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:23:23.084 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:23:23.085 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-p8gb8 0/1 Pending 0 11s 2022-03-10T01:23:33.087 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:23:33.154 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:23:33.155 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-p8gb8 0/1 Pending 0 21s 2022-03-10T01:23:43.157 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:23:43.224 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:23:43.225 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-p8gb8 0/1 Pending 0 31s 2022-03-10T01:23:53.226 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:23:53.376 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:23:53.376 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-p8gb8 0/1 ContainerCreating 0 41s 2022-03-10T01:24:03.380 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:24:03.441 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:24:03.442 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-p8gb8 0/1 ContainerCreating 0 51s 2022-03-10T01:24:13.444 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:24:13.507 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:24:13.507 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-p8gb8 0/1 ContainerCreating 0 61s 2022-03-10T01:24:23.510 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:24:23.579 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:24:23.580 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-p8gb8 0/1 ContainerCreating 0 71s 2022-03-10T01:24:33.584 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:24:33.646 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:24:33.647 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-operator-6b496c8959-p8gb8 1/1 Running 0 81s 2022-03-10T01:24:33.649 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph logs -f rook-ceph-operator-6b496c8959-p8gb8 2022-03-10T01:24:33.656 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:24:33.657 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:24:33.664 INFO:tasks.rook: override: [global] osd crush chooseleaf type = 0 2022-03-10T01:24:33.664 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-10T01:24:33.665 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-10T01:24:33.665 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-10T01:24:33.666 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-10T01:24:33.667 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-10T01:24:33.667 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-10T01:24:33.667 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-10T01:24:33.668 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-10T01:24:33.671 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f - 2022-03-10T01:24:33.749 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:31.914547 I | rookcmd: starting Rook v1.7.2 with arguments '/usr/local/bin/rook ceph operator' 2022-03-10T01:24:33.750 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:31.914630 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule= 2022-03-10T01:24:33.751 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:31.914634 I | cephcmd: starting Rook-Ceph operator 2022-03-10T01:24:33.751 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.069475 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)" 2022-03-10T01:24:33.751 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.076815 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T01:24:33.752 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.078594 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T01:24:33.752 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.081225 I | operator: looking for secret "rook-ceph-admission-controller" 2022-03-10T01:24:33.753 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.082797 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T01:24:33.754 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.085359 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-10T01:24:33.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.085376 I | operator: watching all namespaces for ceph cluster CRs 2022-03-10T01:24:33.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.085434 I | operator: setting up the controller-runtime manager 2022-03-10T01:24:33.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.087247 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations 2022-03-10T01:24:33.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.094905 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T01:24:33.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.493595 I | ceph-cluster-controller: successfully started 2022-03-10T01:24:33.756 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.493685 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T01:24:33.756 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.493707 I | ceph-crashcollector-controller: successfully started 2022-03-10T01:24:33.756 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.493790 I | ceph-block-pool-controller: successfully started 2022-03-10T01:24:33.756 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.493894 I | ceph-object-store-user-controller: successfully started 2022-03-10T01:24:33.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.493965 I | ceph-object-realm-controller: successfully started 2022-03-10T01:24:33.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.494038 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T01:24:33.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.494106 I | ceph-object-zone-controller: successfully started 2022-03-10T01:24:33.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.494283 I | ceph-object-controller: successfully started 2022-03-10T01:24:33.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.494381 I | ceph-file-controller: successfully started 2022-03-10T01:24:33.758 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.494483 I | ceph-nfs-controller: successfully started 2022-03-10T01:24:33.758 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.494569 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T01:24:33.758 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.494662 I | ceph-client-controller: successfully started 2022-03-10T01:24:33.758 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.494731 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T01:24:33.759 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:32.495789 I | operator: starting the controller-runtime manager 2022-03-10T01:24:34.673 INFO:teuthology.orchestra.run.smithi186.stdout:configmap/rook-config-override created 2022-03-10T01:24:34.678 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 1 skipUpgradeChecks: true 2022-03-10T01:24:34.679 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:24:34.679 DEBUG:teuthology.orchestra.run.smithi186:> dd of=cluster.yaml 2022-03-10T01:24:34.686 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f cluster.yaml 2022-03-10T01:24:37.253 INFO:teuthology.orchestra.run.smithi186.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-10T01:24:37.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.265080 I | clusterdisruption-controller: create event from ceph cluster CR 2022-03-10T01:24:37.264 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.265379 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-10T01:24:37.316 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-10T01:24:37.331 DEBUG:teuthology.orchestra.run.smithi186:> kubectl create -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-10T01:24:37.333 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.338348 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:24:37.336 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.339038 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T01:24:37.336 INFO:tasks.rook.operator.smithi186.stdout: ... // 3 identical fields 2022-03-10T01:24:37.336 INFO:tasks.rook.operator.smithi186.stdout: Labels: nil, 2022-03-10T01:24:37.337 INFO:tasks.rook.operator.smithi186.stdout: Placement: nil, 2022-03-10T01:24:37.337 INFO:tasks.rook.operator.smithi186.stdout: Network: v1.NetworkSpec{ 2022-03-10T01:24:37.337 INFO:tasks.rook.operator.smithi186.stdout: Provider: "", 2022-03-10T01:24:37.337 INFO:tasks.rook.operator.smithi186.stdout: Selectors: nil, 2022-03-10T01:24:37.338 INFO:tasks.rook.operator.smithi186.stdout: HostNetwork: false, 2022-03-10T01:24:37.338 INFO:tasks.rook.operator.smithi186.stdout:- IPFamily: "", 2022-03-10T01:24:37.338 INFO:tasks.rook.operator.smithi186.stdout:+ IPFamily: "IPv4", 2022-03-10T01:24:37.338 INFO:tasks.rook.operator.smithi186.stdout: DualStack: false, 2022-03-10T01:24:37.339 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-10T01:24:37.339 INFO:tasks.rook.operator.smithi186.stdout: Resources: nil, 2022-03-10T01:24:37.339 INFO:tasks.rook.operator.smithi186.stdout: PriorityClassNames: nil, 2022-03-10T01:24:37.339 INFO:tasks.rook.operator.smithi186.stdout: ... // 16 identical fields 2022-03-10T01:24:37.339 INFO:tasks.rook.operator.smithi186.stdout: } 2022-03-10T01:24:37.340 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.340980 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-10T01:24:37.348 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.354911 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T01:24:37.374 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.379732 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T01:24:37.379 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.380156 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster 2022-03-10T01:24:37.379 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.383105 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T01:24:37.381 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.386215 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T01:24:37.382 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.388319 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T01:24:37.388 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.393369 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T01:24:37.390 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.396164 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:24:37.395 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.401679 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.4.0" (default) 2022-03-10T01:24:37.580 INFO:teuthology.orchestra.run.smithi186.stdout:deployment.apps/rook-ceph-tools created 2022-03-10T01:24:37.583 INFO:tasks.rook:Waiting for tools container to start 2022-03-10T01:24:37.584 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:24:37.694 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:24:37.694 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 Pending 0 0s 2022-03-10T01:24:37.760 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:37.766004 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0" (default) 2022-03-10T01:24:38.136 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:38.142078 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2" (default) 2022-03-10T01:24:38.335 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:38.341423 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.2.1" (default) 2022-03-10T01:24:38.574 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:38.580113 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1" (default) 2022-03-10T01:24:38.875 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:38.881552 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T01:24:38.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:38.942215 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default) 2022-03-10T01:24:39.136 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:39.142524 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T01:24:39.336 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:39.342549 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T01:24:39.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:39.342576 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.4.0" 2022-03-10T01:24:39.556 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:24:39.561887 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:24:42.700 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:24:42.767 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:24:42.767 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 5s 2022-03-10T01:24:47.773 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:24:47.835 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:24:47.836 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 10s 2022-03-10T01:24:52.838 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:24:52.900 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:24:52.900 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 15s 2022-03-10T01:24:57.903 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:24:57.966 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:24:57.967 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 20s 2022-03-10T01:25:02.969 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:03.027 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:03.027 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 26s 2022-03-10T01:25:08.030 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:08.103 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:08.104 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 31s 2022-03-10T01:25:13.106 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:13.171 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:13.171 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 36s 2022-03-10T01:25:18.173 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:18.510 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:18.510 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 41s 2022-03-10T01:25:23.513 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:23.875 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:23.876 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 46s 2022-03-10T01:25:28.879 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:28.997 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:28.998 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 52s 2022-03-10T01:25:33.989 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:33.995746 I | ceph-csi: Detected ceph CSI image version: "v3.4.0" 2022-03-10T01:25:34.001 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:34.249 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:34.250 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 57s 2022-03-10T01:25:34.250 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.258018 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T01:25:34.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.258211 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:25:34.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.258355 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:25:34.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.258519 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:25:34.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.329092 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:25:34.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.329155 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T01:25:34.463 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.470328 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T01:25:34.516 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.523573 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T01:25:34.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.528346 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T01:25:34.611 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.614152 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:25:34.619 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.626490 I | op-mon: start running mons 2022-03-10T01:25:34.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.627845 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:25:34.665 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.672494 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T01:25:34.720 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.727146 I | op-mon: creating mon secrets for a new cluster 2022-03-10T01:25:34.772 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:34.779896 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T01:25:35.163 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:35.170437 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T01:25:35.562 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:35.569412 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T01:25:35.760 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:35.767675 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-10T01:25:35.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:35.968346 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:25:36.164 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:36.171958 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-10T01:25:36.372 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:36.379216 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:25:36.372 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:36.379249 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T01:25:36.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:36.769817 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0" (default) 2022-03-10T01:25:37.162 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:37.169719 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T01:25:37.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:37.769536 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:25:37.763 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:37.769758 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:25:37.961 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:37.969028 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:25:38.360 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:38.368149 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:25:38.788 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:38.795904 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:25:39.162 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:39.169409 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:25:39.254 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:39.319 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:39.320 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 62s 2022-03-10T01:25:39.562 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:39.569652 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:25:39.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:39.970039 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:25:40.164 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:40.171582 I | op-mon: targeting the mon count 1 2022-03-10T01:25:40.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:40.194841 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-10T01:25:40.188 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:40.194964 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:25:40.361 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:40.368227 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T01:25:40.761 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:40.768782 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:25:40.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:40.969329 I | op-mon: parsing mon endpoints: 2022-03-10T01:25:40.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:40.969362 W | op-mon: ignoring invalid monitor 2022-03-10T01:25:40.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:40.981394 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:25:41.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:41.200516 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:25:41.656 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:41.662685 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:25:41.983 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:41.990625 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T01:25:43.153 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:43.159075 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:25:43.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:43.163448 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:25:43.158 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:43.165787 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T01:25:43.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:43.229095 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:25:43.359 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:43.365572 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:25:43.435 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:43.441653 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:25:44.318 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:44.325273 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T01:25:44.323 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:44.624 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:44.625 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 67s 2022-03-10T01:25:45.429 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:45.425596 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T01:25:45.506 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:45.512809 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T01:25:45.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:45.962132 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-03-10T01:25:45.959 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:45.966998 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T01:25:49.627 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:49.781 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:49.781 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 0/1 ContainerCreating 0 72s 2022-03-10T01:25:54.784 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:25:55.012 INFO:teuthology.orchestra.run.smithi186.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:25:55.013 INFO:teuthology.orchestra.run.smithi186.stdout:rook-ceph-tools-7677f7f4b9-7sxwg 1/1 Running 0 78s 2022-03-10T01:25:55.014 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-10T01:25:55.015 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:25:55.464 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:25:55.464 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:25:55.467+0000 7f4eb4422700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:25:55.464 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:25:55.467+0000 7f4eb4422700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:25:55.464 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-10T01:25:55.467 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-10T01:25:55.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:25:58.448 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:58.456585 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:25:58.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:58.456617 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:25:58.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:58.456631 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:25:58.474 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:58.481877 I | op-mon: parsing mon endpoints: 2022-03-10T01:25:58.474 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:58.481903 W | op-mon: ignoring invalid monitor 2022-03-10T01:25:58.687 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:58.695308 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:25:58.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:58.695537 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:25:58.803 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:58.810814 E | ceph-cluster-controller: failed to get ceph daemons versions, this typically happens during the first cluster initialization. failed to run 'ceph versions'. . unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:25:58.803 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:25:58.803+0000 7fce1855e700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:25:58.804 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:25:58.803+0000 7fce1855e700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:25:58.804 INFO:tasks.rook.operator.smithi186.stdout:[errno 2] RADOS object not found (error connecting to the cluster): exit status 1 2022-03-10T01:25:58.804 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:58.810874 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:25:59.615 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:59.623045 I | op-mon: start running mons 2022-03-10T01:25:59.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:59.715258 I | op-mon: parsing mon endpoints: 2022-03-10T01:25:59.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:59.715278 W | op-mon: ignoring invalid monitor 2022-03-10T01:25:59.767 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:59.775555 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-10T01:25:59.774 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:59.782418 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:25:59.774 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:25:59.782557 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:26:00.603 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:00.610215 I | op-mon: targeting the mon count 1 2022-03-10T01:26:00.609 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:00.617233 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-10T01:26:00.935 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:00.942317 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi186 2022-03-10T01:26:00.935 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:00.942349 I | op-mon: assignmon: mon a assigned to node smithi186 2022-03-10T01:26:00.943 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:00.947559 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-10T01:26:00.946 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:00.952639 I | op-mon: creating mon a 2022-03-10T01:26:01.943 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:01.948884 I | op-mon: mon "a" endpoint is [v2:10.107.13.124:3300,v1:10.107.13.124:6789] 2022-03-10T01:26:03.132 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:03.139847 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-10T01:26:03.132 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:03.140101 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-10T01:26:03.135 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:03.140832 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:-1] 2022-03-10T01:26:03.295 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:03.239408 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:26:03.295 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:03.303480 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:26:03.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:03.365599 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-10T01:26:03.466 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:03.474643 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-10T01:26:04.065 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:04.073503 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:26:04.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:04.073526 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:26:04.487 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:04.495527 I | op-mon: mon a is not yet running 2022-03-10T01:26:04.488 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:04.495579 I | op-mon: mons running: [] 2022-03-10T01:26:05.470 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:26:06.088 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:26:06.089 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:06.091+0000 7fab3b04e700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:26:06.089 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:06.091+0000 7fab3b04e700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:26:06.089 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-10T01:26:06.091 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-10T01:26:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:26:16.095 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:26:16.613 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:26:16.613 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:16.619+0000 7f01a26b4700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:26:16.614 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-10T01:26:16.614 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:16.619+0000 7f01a26b4700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:26:16.615 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-10T01:26:16.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:26:24.705 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:24.713966 I | op-mon: mons running: [a] 2022-03-10T01:26:24.940 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:24.949050 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:26:24.941 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:24.949070 I | op-mon: mons created: 1 2022-03-10T01:26:25.161 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:25.169835 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:26:25.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:25.176385 I | op-mon: mons running: [a] 2022-03-10T01:26:25.395 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:25.403940 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:26:25.395 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:25.403967 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:26:25.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:25.740857 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:26:25.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:25.740918 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:26:26.022 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:26.030965 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:26:26.023 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:26.030990 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:26:26.316 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:26.324830 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:26:26.316 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:26.324854 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:26:26.619 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:26:26.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:26.644170 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:26:26.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:26.644199 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:26:26.894 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:26:26.894 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:26.899+0000 7f8021a94700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:26:26.894 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:26.899+0000 7f8021a94700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:26:26.894 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-10T01:26:26.898 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-10T01:26:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:26:27.034 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:27.043077 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:26:27.035 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:27.043108 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T01:26:27.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:27.268125 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T01:26:27.260 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:27.268170 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T01:26:27.564 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:27.572649 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T01:26:27.841 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:27.849849 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T01:26:28.211 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:28.220189 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T01:26:28.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:28.604347 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T01:26:28.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:28.604367 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T01:26:28.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:28.904760 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T01:26:29.116 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:29.125004 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T01:26:29.117 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:29.125027 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T01:26:29.117 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:29.125031 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T01:26:29.412 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:29.420776 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T01:26:29.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:29.433917 I | op-mgr: start running mgr 2022-03-10T01:26:29.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:29.433943 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T01:26:30.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:30.267343 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:26:36.900 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:26:37.387 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:26:37.387 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:37.391+0000 7f7953261700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:26:37.387 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:37.391+0000 7f7953261700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:26:37.387 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-10T01:26:37.389 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-10T01:26:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:26:41.857 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:41.866051 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-10T01:26:41.861 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:41.868658 I | op-mgr: setting services to point to mgr "a" 2022-03-10T01:26:41.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:41.972201 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T01:26:41.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:41.972228 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-10T01:26:41.964 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:41.972238 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T01:26:41.966 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:41.975791 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T01:26:41.971 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:41.980208 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-10T01:26:41.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:41.980318 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T01:26:41.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:41.980329 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T01:26:42.213 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.222078 W | cephclient: failed to enable mgr module "rook". trying again... 2022-03-10T01:26:42.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.266084 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T01:26:42.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.266115 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:26:42.270 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.276827 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-10T01:26:42.270 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.276858 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-10T01:26:42.270 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.276877 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-10T01:26:42.303 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.312127 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T01:26:42.303 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.312145 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:26:42.306 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.314723 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Normal:ReconcileSucceeded:successfully configured CephCluster "rook-ceph/rook-ceph" 2022-03-10T01:26:42.307 INFO:tasks.rook.operator.smithi186.stdout:I0310 01:26:42.315330 7 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:26:42.311 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.317876 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:26:42.363 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.372301 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:26:42.376 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.385509 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:26:42.387 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.396844 I | op-mgr: successful modules: prometheus 2022-03-10T01:26:42.638 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:42.647509 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-10T01:26:43.567 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:43.576749 I | op-mgr: successful modules: dashboard 2022-03-10T01:26:47.392 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:26:47.672 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:26:47.672 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:47.675+0000 7f5ab6dac700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:26:47.672 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:47.675+0000 7f5ab6dac700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:26:47.672 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-10T01:26:47.674 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-10T01:26:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:26:47.919 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:47.928072 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-10T01:26:49.074 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.083801 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T01:26:49.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.212979 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:26:49.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.213018 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:26:49.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.213039 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:26:49.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.240882 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:26:49.296 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.306019 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:26:49.297 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.306239 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:26:49.523 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.532767 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:26:49.603 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.613045 I | op-mon: start running mons 2022-03-10T01:26:49.631 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.640164 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:26:49.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.679090 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:26:49.676 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.685592 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:26:49.676 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:49.685824 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:26:50.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:50.816576 I | op-mon: targeting the mon count 1 2022-03-10T01:26:50.815 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:50.824273 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:26:51.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.084323 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:26:51.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.084355 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:26:51.118 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.127835 I | cephclient: command failed for set rook backend. trying again... 2022-03-10T01:26:51.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.300940 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:26:51.292 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.300974 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:26:51.500 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.509368 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:26:51.500 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.509400 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:26:51.727 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.736251 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:26:51.727 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.736298 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:26:51.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.983849 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:26:51.975 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:51.983880 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T01:26:52.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:52.200643 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T01:26:52.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:52.200672 I | op-mon: checking for basic quorum with existing mons 2022-03-10T01:26:52.313 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:52.322936 I | op-mon: mon "a" endpoint is [v2:10.107.13.124:3300,v1:10.107.13.124:6789] 2022-03-10T01:26:52.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:52.347128 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:26:52.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:52.816615 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:26:52.814 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:52.816883 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:26:53.216 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:53.225113 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-10T01:26:53.218 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:53.226360 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T01:26:53.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:53.233874 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T01:26:53.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:53.233900 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:26:53.420 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:53.429824 I | op-mon: mons running: [a] 2022-03-10T01:26:53.633 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:53.642481 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:26:53.633 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:53.642507 I | op-mon: mons created: 1 2022-03-10T01:26:54.015 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:54.021001 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:26:54.021 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:54.031268 I | op-mon: mons running: [a] 2022-03-10T01:26:54.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:54.244552 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:26:54.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:54.244597 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T01:26:54.506 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:54.515702 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T01:26:54.745 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:54.754576 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T01:26:54.979 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:54.988173 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T01:26:55.229 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:55.238238 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T01:26:55.229 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:55.238259 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T01:26:55.610 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:55.619310 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T01:26:55.841 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:55.850632 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T01:26:55.842 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:55.850664 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T01:26:55.842 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:55.850671 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T01:26:56.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:56.084210 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T01:26:56.100 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:56.110274 I | op-mgr: start running mgr 2022-03-10T01:26:56.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:56.110306 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T01:26:56.435 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:56.444412 I | cephclient: command failed for set rook backend. trying again... 2022-03-10T01:26:56.650 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:56.659682 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T01:26:56.660 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:56.669884 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T01:26:56.662 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:56.672011 I | op-mgr: setting services to point to mgr "a" 2022-03-10T01:26:57.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:57.455269 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T01:26:57.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:57.455298 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-10T01:26:57.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:57.455309 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T01:26:57.494 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:57.503143 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T01:26:57.607 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:57.616478 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-10T01:26:57.608 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:57.616502 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T01:26:57.608 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:57.616508 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T01:26:57.677 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:26:57.956 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:26:57.957 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:57.963+0000 7f90eb111700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:26:57.957 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:26:57.963+0000 7f90eb111700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:26:57.957 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-10T01:26:57.959 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-10T01:26:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:26:58.276 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:58.285755 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T01:26:58.277 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:58.285774 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:26:58.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:58.346601 I | op-mgr: successful modules: prometheus 2022-03-10T01:26:58.340 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:58.347397 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T01:26:58.343 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:58.351877 I | op-mgr: successful modules: dashboard 2022-03-10T01:26:58.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:58.622313 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-10T01:26:58.734 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:58.744145 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-10T01:26:59.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:26:59.692542 I | cephclient: command failed for set rook backend. trying again... 2022-03-10T01:27:01.745 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:01.754411 I | cephclient: command failed for set rook backend. trying again... 2022-03-10T01:27:04.055 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:04.065414 I | op-mgr: successful modules: balancer 2022-03-10T01:27:04.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:04.110831 I | op-mgr: successful modules: balancer 2022-03-10T01:27:05.108 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:05.117472 I | cephclient: action set rook backend succeeded on attempt 1 2022-03-10T01:27:05.108 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:05.117509 I | op-mgr: successful modules: orchestrator modules 2022-03-10T01:27:07.148 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:07.157865 I | cephclient: action set rook backend succeeded on attempt 3 2022-03-10T01:27:07.149 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:07.157905 I | op-mgr: successful modules: orchestrator modules 2022-03-10T01:27:07.963 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:27:08.251 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:27:08.252 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:27:08.255+0000 7f1dc23ed700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:27:08.252 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:27:08.255+0000 7f1dc23ed700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:27:08.253 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-10T01:27:08.255 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-10T01:27:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:27:18.259 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:27:18.542 INFO:teuthology.orchestra.run.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-10T01:27:18.543 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:27:18.547+0000 7f484d725700 -1 failed for service _ceph-mon._tcp 2022-03-10T01:27:18.543 INFO:teuthology.orchestra.run.smithi186.stderr:2022-03-10T01:27:18.547+0000 7f484d725700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-10T01:27:18.543 INFO:teuthology.orchestra.run.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-03-10T01:27:18.546 INFO:teuthology.orchestra.run.smithi186.stderr:command terminated with exit code 1 2022-03-10T01:27:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:27:28.550 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status -f json 2022-03-10T01:27:29.128 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:27:29.128 INFO:teuthology.orchestra.run.smithi186.stdout:{"available": true, "backend": "rook"} 2022-03-10T01:27:29.148 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-10T01:27:29.148 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-10T01:27:29.789 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch apply osd --all-available-devices 2022-03-10T01:27:30.443 INFO:teuthology.orchestra.run.smithi186.stdout:['Success'] 2022-03-10T01:27:30.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:30.461187 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T01:27:30.454 INFO:tasks.rook.operator.smithi186.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-10T01:27:30.454 INFO:tasks.rook.operator.smithi186.stdout: Storage: v1.StorageScopeSpec{ 2022-03-10T01:27:30.454 INFO:tasks.rook.operator.smithi186.stdout: ... // 3 identical fields 2022-03-10T01:27:30.454 INFO:tasks.rook.operator.smithi186.stdout: Config: nil, 2022-03-10T01:27:30.455 INFO:tasks.rook.operator.smithi186.stdout: Selection: {}, 2022-03-10T01:27:30.455 INFO:tasks.rook.operator.smithi186.stdout:- StorageClassDeviceSets: nil, 2022-03-10T01:27:30.455 INFO:tasks.rook.operator.smithi186.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T01:27:30.455 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-10T01:27:30.456 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-1", 2022-03-10T01:27:30.456 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-10T01:27:30.456 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:27:30.456 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-10T01:27:30.456 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-10T01:27:30.457 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-2", 2022-03-10T01:27:30.457 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-10T01:27:30.457 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:27:30.457 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-10T01:27:30.457 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-10T01:27:30.458 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-3", 2022-03-10T01:27:30.458 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-10T01:27:30.458 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:27:30.458 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-10T01:27:30.459 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-10T01:27:30.459 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-4", 2022-03-10T01:27:30.459 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-10T01:27:30.459 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:27:30.460 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-10T01:27:30.460 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-10T01:27:30.460 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-10T01:27:30.460 INFO:tasks.rook.operator.smithi186.stdout: Annotations: nil, 2022-03-10T01:27:30.461 INFO:tasks.rook.operator.smithi186.stdout: Labels: nil, 2022-03-10T01:27:30.461 INFO:tasks.rook.operator.smithi186.stdout: ... // 20 identical fields 2022-03-10T01:27:30.461 INFO:tasks.rook.operator.smithi186.stdout: } 2022-03-10T01:27:30.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:30.461269 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:27:30.468 INFO:tasks.rook:Waiting for 4 OSDs 2022-03-10T01:27:30.468 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:27:30.507 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:30.517177 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:27:30.523 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:30.533430 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:27:31.057 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:27:31.073 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:27:31.073 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:27:36.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:36.905310 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:27:36.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:36.905351 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:27:36.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:36.905373 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:27:36.905 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:36.915712 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:27:37.338 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:37.348858 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:27:37.342 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:37.351422 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:27:37.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:37.804302 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:27:37.881 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:37.891698 I | op-mon: start running mons 2022-03-10T01:27:37.899 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:37.909924 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:27:37.928 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:37.938403 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:27:37.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:37.946838 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:27:37.939 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:37.947107 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:27:38.937 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:38.947449 I | op-mon: targeting the mon count 1 2022-03-10T01:27:38.949 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:38.959426 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-10T01:27:39.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:39.960358 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:27:40.056 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:40.066641 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:27:40.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:40.098709 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:27:40.114 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:40.125058 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-10T01:27:40.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:40.131237 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T01:27:41.075 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:27:41.587 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:27:41.611 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:27:41.611 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:27:42.127 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:42.137426 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-10T01:27:49.448 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:49.448784 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:27:49.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:49.448805 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:27:49.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:49.448812 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:27:49.657 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:49.667122 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:27:49.869 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:49.878694 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:27:49.869 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:49.878922 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:27:50.283 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:50.293896 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:27:50.456 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:50.466718 I | op-mon: start running mons 2022-03-10T01:27:50.548 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:50.558923 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:27:50.597 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:50.607609 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:27:50.605 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:50.615965 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:27:50.607 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:50.616662 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:27:51.309 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:51.319630 I | op-mon: targeting the mon count 1 2022-03-10T01:27:51.314 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:51.324704 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:27:51.613 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:27:51.668 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:51.679266 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:27:51.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:51.679294 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:27:52.025 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:52.035986 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:27:52.025 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:52.036015 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:27:52.108 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:27:52.127 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:27:52.127 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:27:52.369 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:52.379392 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:27:52.369 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:52.379422 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:27:52.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:52.709941 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:27:52.700 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:52.709969 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:27:53.037 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.047864 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:27:53.037 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.047893 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T01:27:53.383 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.393873 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T01:27:53.383 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.393907 I | op-mon: checking for basic quorum with existing mons 2022-03-10T01:27:53.556 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.567051 I | op-mon: mon "a" endpoint is [v2:10.107.13.124:3300,v1:10.107.13.124:6789] 2022-03-10T01:27:53.588 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.599108 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:27:53.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.606723 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:27:53.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.606927 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:27:53.800 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.810544 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T01:27:53.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.819633 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T01:27:53.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:53.819666 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:27:53.995 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:54.005762 I | op-mon: mons running: [a] 2022-03-10T01:27:54.409 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:54.419769 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:27:54.409 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:54.419816 I | op-mon: mons created: 1 2022-03-10T01:27:54.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:54.824113 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:27:54.818 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:54.829767 I | op-mon: mons running: [a] 2022-03-10T01:27:55.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:55.234702 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:27:55.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:55.234748 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T01:27:55.641 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:55.652371 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T01:27:56.054 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:56.064933 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T01:27:56.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:56.515055 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T01:27:56.983 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:56.993785 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T01:27:56.983 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:56.993805 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T01:27:57.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:57.418686 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T01:27:57.821 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:57.831813 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T01:27:57.821 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:57.831852 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T01:27:57.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:57.831861 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T01:27:58.233 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.243722 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T01:27:58.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.301999 I | op-mgr: start running mgr 2022-03-10T01:27:58.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.302041 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T01:27:58.766 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.776780 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T01:27:58.774 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.785610 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T01:27:58.777 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.788034 I | op-mgr: setting services to point to mgr "a" 2022-03-10T01:27:58.857 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.867969 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T01:27:58.858 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.868014 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T01:27:58.934 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.944857 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T01:27:58.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:58.979060 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-1" 2022-03-10T01:27:59.172 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.183250 I | op-osd: successfully provisioned PVC "smithi186-lv-1-data-0c49qw" 2022-03-10T01:27:59.173 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.183345 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-2" 2022-03-10T01:27:59.434 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.445240 I | op-osd: successfully provisioned PVC "smithi186-lv-2-data-02p4st" 2022-03-10T01:27:59.434 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.445284 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-3" 2022-03-10T01:27:59.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.622482 I | op-osd: successfully provisioned PVC "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:27:59.612 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.622521 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-4" 2022-03-10T01:27:59.736 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.747606 I | op-mgr: successful modules: dashboard 2022-03-10T01:27:59.737 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.747733 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T01:27:59.770 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.781541 I | op-osd: successfully provisioned PVC "smithi186-lv-4-data-0g6blf" 2022-03-10T01:27:59.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:27:59.806137 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0c49qw" 2022-03-10T01:28:00.107 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:00.117837 I | op-mgr: successful modules: balancer 2022-03-10T01:28:00.203 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:00.214306 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-1-data-0c49qw" 2022-03-10T01:28:00.203 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:00.214331 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-02p4st" 2022-03-10T01:28:00.599 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:00.610424 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-2-data-02p4st" 2022-03-10T01:28:00.600 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:00.610453 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:28:00.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:00.979151 I | op-mgr: successful modules: prometheus 2022-03-10T01:28:01.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:01.015686 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:28:01.005 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:01.015711 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0g6blf" 2022-03-10T01:28:01.335 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:01.346228 I | op-mgr: successful modules: orchestrator modules 2022-03-10T01:28:01.909 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:01.920229 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-4-data-0g6blf" 2022-03-10T01:28:01.910 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:01.920260 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T01:28:01.910 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:01.920268 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T01:28:02.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:02.114553 I | op-osd: OSD orchestration status for PVC smithi186-lv-1-data-0c49qw is "starting" 2022-03-10T01:28:02.105 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:02.114593 I | op-osd: OSD orchestration status for PVC smithi186-lv-2-data-02p4st is "starting" 2022-03-10T01:28:02.105 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:02.114604 I | op-osd: OSD orchestration status for PVC smithi186-lv-3-data-0b5pdc is "starting" 2022-03-10T01:28:02.105 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:02.114612 I | op-osd: OSD orchestration status for PVC smithi186-lv-4-data-0g6blf is "starting" 2022-03-10T01:28:02.129 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:28:03.275 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:28:03.293 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:28:03.293 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:28:13.294 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:28:14.097 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:28:14.119 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:28:14.120 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:28:18.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:18.701538 I | op-osd: OSD orchestration status for node smithi186-lv-2-data-02p4st is "orchestrating" 2022-03-10T01:28:18.986 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:18.997560 I | op-osd: OSD orchestration status for node smithi186-lv-4-data-0g6blf is "orchestrating" 2022-03-10T01:28:18.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:19.010162 I | op-osd: OSD orchestration status for node smithi186-lv-3-data-0b5pdc is "orchestrating" 2022-03-10T01:28:19.367 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:19.378536 I | op-osd: OSD orchestration status for PVC smithi186-lv-4-data-0g6blf is "orchestrating" 2022-03-10T01:28:19.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:19.407820 I | op-osd: OSD orchestration status for PVC smithi186-lv-2-data-02p4st is "orchestrating" 2022-03-10T01:28:19.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:19.413615 I | op-osd: OSD orchestration status for PVC smithi186-lv-3-data-0b5pdc is "orchestrating" 2022-03-10T01:28:24.121 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:28:24.622 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:28:24.638 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":9,"num_osds":3,"num_up_osds":0,"osd_up_since":0,"num_in_osds":3,"osd_in_since":1646875702,"num_remapped_pgs":0} 2022-03-10T01:28:24.639 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:28:25.629 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:25.640787 I | op-osd: OSD orchestration status for node smithi186-lv-1-data-0c49qw is "orchestrating" 2022-03-10T01:28:25.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:25.985817 I | op-osd: OSD orchestration status for PVC smithi186-lv-1-data-0c49qw is "orchestrating" 2022-03-10T01:28:26.454 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:26.465400 I | op-osd: OSD orchestration status for PVC smithi186-lv-4-data-0g6blf is "completed" 2022-03-10T01:28:26.454 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:26.465420 I | op-osd: creating OSD 0 on PVC "smithi186-lv-4-data-0g6blf" 2022-03-10T01:28:26.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:26.465429 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0g6blf" 2022-03-10T01:28:27.389 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:27.400138 I | op-osd: OSD orchestration status for PVC smithi186-lv-2-data-02p4st is "completed" 2022-03-10T01:28:27.392 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:27.400572 I | op-osd: creating OSD 2 on PVC "smithi186-lv-2-data-02p4st" 2022-03-10T01:28:27.392 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:27.400609 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-02p4st" 2022-03-10T01:28:27.764 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:27.771266 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:28:29.431 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:29.435749 I | op-osd: OSD orchestration status for PVC smithi186-lv-3-data-0b5pdc is "completed" 2022-03-10T01:28:29.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:29.435781 I | op-osd: creating OSD 1 on PVC "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:28:29.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:29.435800 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:28:34.284 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:34.293812 I | op-osd: OSD orchestration status for PVC smithi186-lv-1-data-0c49qw is "completed" 2022-03-10T01:28:34.284 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:34.293842 I | op-osd: creating OSD 3 on PVC "smithi186-lv-1-data-0c49qw" 2022-03-10T01:28:34.285 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:34.293854 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0c49qw" 2022-03-10T01:28:34.640 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:28:35.799 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:28:35.816 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646875707,"num_remapped_pgs":0} 2022-03-10T01:28:35.816 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:28:36.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:36.291736 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T01:28:36.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:28:36.291764 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:28:45.818 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:28:46.465 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:28:46.482 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646875707,"num_remapped_pgs":0} 2022-03-10T01:28:46.483 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:28:56.484 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:28:57.041 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:28:57.063 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646875707,"num_remapped_pgs":0} 2022-03-10T01:28:57.063 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:29:07.064 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:29:07.652 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:07.671 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1646875707,"num_remapped_pgs":0} 2022-03-10T01:29:07.671 INFO:tasks.rook: have 0/4 OSDs 2022-03-10T01:29:17.672 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd stat -f json 2022-03-10T01:29:18.240 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:18.259 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":16,"num_osds":4,"num_up_osds":4,"osd_up_since":1646875754,"num_in_osds":4,"osd_in_since":1646875707,"num_remapped_pgs":0} 2022-03-10T01:29:18.260 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-10T01:29:18.260 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- cat /etc/ceph/ceph.conf 2022-03-10T01:29:18.451 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2022-03-10T01:29:18.451 INFO:teuthology.orchestra.run.smithi186.stdout:mon_host = 10.107.13.124:6789 2022-03-10T01:29:18.452 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:18.452 INFO:teuthology.orchestra.run.smithi186.stdout:[client.admin] 2022-03-10T01:29:18.452 INFO:teuthology.orchestra.run.smithi186.stdout:keyring = /etc/ceph/keyring 2022-03-10T01:29:18.455 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- cat /etc/ceph/keyring 2022-03-10T01:29:18.603 INFO:teuthology.orchestra.run.smithi186.stdout:[client.admin] 2022-03-10T01:29:18.603 INFO:teuthology.orchestra.run.smithi186.stdout:key = AQCOUyliCkpDKxAAniJhmETCdXka7ce++emO3Q== 2022-03-10T01:29:18.608 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2022-03-10T01:29:18.617 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:29:18.617 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T01:29:18.626 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:29:18.627 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/keyring 2022-03-10T01:29:18.637 INFO:tasks.rook:Setting up client nodes... 2022-03-10T01:29:18.637 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T01:29:19.380 INFO:teuthology.orchestra.run.smithi186.stdout:[client.a] 2022-03-10T01:29:19.381 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQBvVCliXrMwFBAAvgTwROasYkTT0MVXM3k/Dg== 2022-03-10T01:29:19.402 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:29:19.402 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-10T01:29:19.403 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-10T01:29:19.419 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd dump --format=json 2022-03-10T01:29:19.929 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:19.930 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":16,"fsid":"af064395-2045-4803-95db-56693404613d","created":"2022-03-10T01:26:22.338175+0000","modified":"2022-03-10T01:29:17.748586+0000","last_up_change":"2022-03-10T01:29:14.215446+0000","last_in_change":"2022-03-10T01:28:27.633971+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T01:29:15.089310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f44e9515-df3e-4229-86b5-61c9cb824157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6800","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6801","nonce":3255230099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6802","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6803","nonce":3255230099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6806","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6807","nonce":3255230099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6804","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6805","nonce":3255230099}]},"public_addr":"10.253.202.150:6801/3255230099","cluster_addr":"10.253.202.150:6803/3255230099","heartbeat_back_addr":"10.253.202.150:6807/3255230099","heartbeat_front_addr":"10.253.202.150:6805/3255230099","state":["exists","up"]},{"osd":1,"uuid":"ab07ab21-4780-48a3-a1c2-3194be7de825","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6800","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6801","nonce":1420198069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6802","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6803","nonce":1420198069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6806","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6807","nonce":1420198069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6804","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6805","nonce":1420198069}]},"public_addr":"10.253.202.153:6801/1420198069","cluster_addr":"10.253.202.153:6803/1420198069","heartbeat_back_addr":"10.253.202.153:6807/1420198069","heartbeat_front_addr":"10.253.202.153:6805/1420198069","state":["exists","up"]},{"osd":2,"uuid":"2e45cf88-61bc-42ef-9e5d-7594940c391b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6800","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6801","nonce":4294790136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6802","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6803","nonce":4294790136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6806","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6807","nonce":4294790136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6804","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6805","nonce":4294790136}]},"public_addr":"10.253.202.152:6801/4294790136","cluster_addr":"10.253.202.152:6803/4294790136","heartbeat_back_addr":"10.253.202.152:6807/4294790136","heartbeat_front_addr":"10.253.202.152:6805/4294790136","state":["exists","up"]},{"osd":3,"uuid":"25b1e349-8b33-4c5a-8730-e0998262aff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6800","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6801","nonce":3174637262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6802","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6803","nonce":3174637262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6806","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6807","nonce":3174637262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6804","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6805","nonce":3174637262}]},"public_addr":"10.253.202.154:6801/3174637262","cluster_addr":"10.253.202.154:6803/3174637262","heartbeat_back_addr":"10.253.202.154:6807/3174637262","heartbeat_front_addr":"10.253.202.154:6805/3174637262","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:12.262693+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:11.985062+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:12.021599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:12.231117+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T01:29:19.949 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-10T01:29:15.089310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-10T01:29:19.950 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd pool get .mgr pg_num 2022-03-10T01:29:20.477 INFO:teuthology.orchestra.run.smithi186.stdout:pg_num: 1 2022-03-10T01:29:20.499 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-10T01:29:20.500 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-10T01:29:20.501 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph mgr dump --format=json 2022-03-10T01:29:21.083 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:21.102 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":15,"active_gid":4175,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.141:6800","nonce":2308907035},{"type":"v1","addr":"10.253.202.141:6801","nonce":2308907035}]},"active_addr":"10.253.202.141:6801/2308907035","active_change":"2022-03-10T01:27:00.020224+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.253.202.141:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.253.202.141:0","nonce":3160282409}]},{"addrvec":[{"type":"v2","addr":"10.253.202.141:0","nonce":285430760}]},{"addrvec":[{"type":"v2","addr":"10.253.202.141:0","nonce":2757931494}]},{"addrvec":[{"type":"v2","addr":"10.253.202.141:0","nonce":2284680946}]}]}} 2022-03-10T01:29:21.104 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-10T01:29:21.105 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-10T01:29:21.105 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd dump --format=json 2022-03-10T01:29:21.626 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:21.627 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":16,"fsid":"af064395-2045-4803-95db-56693404613d","created":"2022-03-10T01:26:22.338175+0000","modified":"2022-03-10T01:29:17.748586+0000","last_up_change":"2022-03-10T01:29:14.215446+0000","last_in_change":"2022-03-10T01:28:27.633971+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T01:29:15.089310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f44e9515-df3e-4229-86b5-61c9cb824157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6800","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6801","nonce":3255230099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6802","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6803","nonce":3255230099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6806","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6807","nonce":3255230099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6804","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6805","nonce":3255230099}]},"public_addr":"10.253.202.150:6801/3255230099","cluster_addr":"10.253.202.150:6803/3255230099","heartbeat_back_addr":"10.253.202.150:6807/3255230099","heartbeat_front_addr":"10.253.202.150:6805/3255230099","state":["exists","up"]},{"osd":1,"uuid":"ab07ab21-4780-48a3-a1c2-3194be7de825","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6800","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6801","nonce":1420198069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6802","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6803","nonce":1420198069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6806","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6807","nonce":1420198069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6804","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6805","nonce":1420198069}]},"public_addr":"10.253.202.153:6801/1420198069","cluster_addr":"10.253.202.153:6803/1420198069","heartbeat_back_addr":"10.253.202.153:6807/1420198069","heartbeat_front_addr":"10.253.202.153:6805/1420198069","state":["exists","up"]},{"osd":2,"uuid":"2e45cf88-61bc-42ef-9e5d-7594940c391b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6800","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6801","nonce":4294790136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6802","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6803","nonce":4294790136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6806","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6807","nonce":4294790136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6804","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6805","nonce":4294790136}]},"public_addr":"10.253.202.152:6801/4294790136","cluster_addr":"10.253.202.152:6803/4294790136","heartbeat_back_addr":"10.253.202.152:6807/4294790136","heartbeat_front_addr":"10.253.202.152:6805/4294790136","state":["exists","up"]},{"osd":3,"uuid":"25b1e349-8b33-4c5a-8730-e0998262aff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6800","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6801","nonce":3174637262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6802","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6803","nonce":3174637262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6806","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6807","nonce":3174637262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6804","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6805","nonce":3174637262}]},"public_addr":"10.253.202.154:6801/3174637262","cluster_addr":"10.253.202.154:6803/3174637262","heartbeat_back_addr":"10.253.202.154:6807/3174637262","heartbeat_front_addr":"10.253.202.154:6805/3174637262","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:12.262693+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:11.985062+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:12.021599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:12.231117+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T01:29:21.649 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-10T01:29:21.649 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd dump --format=json 2022-03-10T01:29:22.158 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:22.159 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":16,"fsid":"af064395-2045-4803-95db-56693404613d","created":"2022-03-10T01:26:22.338175+0000","modified":"2022-03-10T01:29:17.748586+0000","last_up_change":"2022-03-10T01:29:14.215446+0000","last_in_change":"2022-03-10T01:28:27.633971+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T01:29:15.089310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f44e9515-df3e-4229-86b5-61c9cb824157","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6800","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6801","nonce":3255230099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6802","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6803","nonce":3255230099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6806","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6807","nonce":3255230099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.150:6804","nonce":3255230099},{"type":"v1","addr":"10.253.202.150:6805","nonce":3255230099}]},"public_addr":"10.253.202.150:6801/3255230099","cluster_addr":"10.253.202.150:6803/3255230099","heartbeat_back_addr":"10.253.202.150:6807/3255230099","heartbeat_front_addr":"10.253.202.150:6805/3255230099","state":["exists","up"]},{"osd":1,"uuid":"ab07ab21-4780-48a3-a1c2-3194be7de825","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6800","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6801","nonce":1420198069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6802","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6803","nonce":1420198069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6806","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6807","nonce":1420198069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.153:6804","nonce":1420198069},{"type":"v1","addr":"10.253.202.153:6805","nonce":1420198069}]},"public_addr":"10.253.202.153:6801/1420198069","cluster_addr":"10.253.202.153:6803/1420198069","heartbeat_back_addr":"10.253.202.153:6807/1420198069","heartbeat_front_addr":"10.253.202.153:6805/1420198069","state":["exists","up"]},{"osd":2,"uuid":"2e45cf88-61bc-42ef-9e5d-7594940c391b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6800","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6801","nonce":4294790136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6802","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6803","nonce":4294790136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6806","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6807","nonce":4294790136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.152:6804","nonce":4294790136},{"type":"v1","addr":"10.253.202.152:6805","nonce":4294790136}]},"public_addr":"10.253.202.152:6801/4294790136","cluster_addr":"10.253.202.152:6803/4294790136","heartbeat_back_addr":"10.253.202.152:6807/4294790136","heartbeat_front_addr":"10.253.202.152:6805/4294790136","state":["exists","up"]},{"osd":3,"uuid":"25b1e349-8b33-4c5a-8730-e0998262aff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6800","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6801","nonce":3174637262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6802","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6803","nonce":3174637262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6806","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6807","nonce":3174637262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.253.202.154:6804","nonce":3174637262},{"type":"v1","addr":"10.253.202.154:6805","nonce":3174637262}]},"public_addr":"10.253.202.154:6801/3174637262","cluster_addr":"10.253.202.154:6803/3174637262","heartbeat_back_addr":"10.253.202.154:6807/3174637262","heartbeat_front_addr":"10.253.202.154:6805/3174637262","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:12.262693+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:11.985062+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:12.021599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:29:12.231117+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T01:29:22.179 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph tell osd.0 flush_pg_stats 2022-03-10T01:29:22.180 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph tell osd.1 flush_pg_stats 2022-03-10T01:29:22.181 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph tell osd.2 flush_pg_stats 2022-03-10T01:29:22.181 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph tell osd.3 flush_pg_stats 2022-03-10T01:29:22.585 INFO:teuthology.orchestra.run.smithi186.stdout:55834574852 2022-03-10T01:29:22.586 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd last-stat-seq osd.3 2022-03-10T01:29:22.619 INFO:teuthology.orchestra.run.smithi186.stdout:55834574852 2022-03-10T01:29:22.620 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd last-stat-seq osd.2 2022-03-10T01:29:22.668 INFO:teuthology.orchestra.run.smithi186.stdout:55834574851 2022-03-10T01:29:22.669 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd last-stat-seq osd.1 2022-03-10T01:29:22.761 INFO:teuthology.orchestra.run.smithi186.stdout:55834574852 2022-03-10T01:29:22.762 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd last-stat-seq osd.0 2022-03-10T01:29:23.149 INFO:teuthology.orchestra.run.smithi186.stdout:55834574851 2022-03-10T01:29:23.168 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.2 2022-03-10T01:29:23.175 INFO:teuthology.orchestra.run.smithi186.stdout:55834574851 2022-03-10T01:29:23.194 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.3 2022-03-10T01:29:23.230 INFO:teuthology.orchestra.run.smithi186.stdout:55834574850 2022-03-10T01:29:23.248 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.1 2022-03-10T01:29:23.288 INFO:teuthology.orchestra.run.smithi186.stdout:55834574851 2022-03-10T01:29:23.306 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2022-03-10T01:29:24.169 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd last-stat-seq osd.2 2022-03-10T01:29:24.195 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd last-stat-seq osd.3 2022-03-10T01:29:24.249 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd last-stat-seq osd.1 2022-03-10T01:29:24.308 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd last-stat-seq osd.0 2022-03-10T01:29:24.727 INFO:teuthology.orchestra.run.smithi186.stdout:55834574852 2022-03-10T01:29:24.750 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2022-03-10T01:29:24.751 DEBUG:teuthology.parallel:result is None 2022-03-10T01:29:24.805 INFO:teuthology.orchestra.run.smithi186.stdout:55834574852 2022-03-10T01:29:24.826 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2022-03-10T01:29:24.826 DEBUG:teuthology.parallel:result is None 2022-03-10T01:29:24.830 INFO:teuthology.orchestra.run.smithi186.stdout:55834574852 2022-03-10T01:29:24.849 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2022-03-10T01:29:24.850 DEBUG:teuthology.parallel:result is None 2022-03-10T01:29:24.888 INFO:teuthology.orchestra.run.smithi186.stdout:55834574851 2022-03-10T01:29:24.908 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.1 2022-03-10T01:29:24.908 DEBUG:teuthology.parallel:result is None 2022-03-10T01:29:24.909 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-10T01:29:24.909 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph pg dump --format=json 2022-03-10T01:29:25.409 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:25.413 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2022-03-10T01:29:25.436 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2022-03-10T01:29:24.585218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":22564,"kb_used_data":1924,"kb_used_omap":0,"kb_used_meta":20352,"kb_avail":374958044,"statfs":{"total":383980142592,"available":383957037056,"internally_reserved":0,"allocated":1970176,"data_stored":1524332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20840448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000819"},"pg_stats":[{"pgid":"1.0","version":"16'97","reported_seq":137,"reported_epoch":16,"state":"active+clean","last_fresh":"2022-03-10T01:29:20.952648+0000","last_change":"2022-03-10T01:29:16.830085+0000","last_active":"2022-03-10T01:29:20.952648+0000","last_peered":"2022-03-10T01:29:20.952648+0000","last_clean":"2022-03-10T01:29:20.952648+0000","last_became_active":"2022-03-10T01:29:16.828298+0000","last_became_peered":"2022-03-10T01:29:16.828298+0000","last_unstale":"2022-03-10T01:29:20.952648+0000","last_undegraded":"2022-03-10T01:29:20.952648+0000","last_fullsized":"2022-03-10T01:29:20.952648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-10T01:29:15.494246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-10T01:29:15.494246+0000","last_clean_scrub_stamp":"2022-03-10T01:29:15.494246+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-11T09:48:07.548181+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":4944,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740208,"statfs":{"total":95995035648,"available":95989972992,"internally_reserved":0,"allocated":139264,"data_stored":30866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5788,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739364,"statfs":{"total":95995035648,"available":95989108736,"internally_reserved":0,"allocated":610304,"data_stored":497822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5916,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739236,"statfs":{"total":95995035648,"available":95988977664,"internally_reserved":0,"allocated":610304,"data_stored":497822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5916,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739236,"statfs":{"total":95995035648,"available":95988977664,"internally_reserved":0,"allocated":610304,"data_stored":497822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-10T01:29:25.437 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph pg dump --format=json 2022-03-10T01:29:25.963 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:25.966 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2022-03-10T01:29:25.981 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2022-03-10T01:29:24.585218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":22564,"kb_used_data":1924,"kb_used_omap":0,"kb_used_meta":20352,"kb_avail":374958044,"statfs":{"total":383980142592,"available":383957037056,"internally_reserved":0,"allocated":1970176,"data_stored":1524332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20840448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000819"},"pg_stats":[{"pgid":"1.0","version":"16'97","reported_seq":137,"reported_epoch":16,"state":"active+clean","last_fresh":"2022-03-10T01:29:20.952648+0000","last_change":"2022-03-10T01:29:16.830085+0000","last_active":"2022-03-10T01:29:20.952648+0000","last_peered":"2022-03-10T01:29:20.952648+0000","last_clean":"2022-03-10T01:29:20.952648+0000","last_became_active":"2022-03-10T01:29:16.828298+0000","last_became_peered":"2022-03-10T01:29:16.828298+0000","last_unstale":"2022-03-10T01:29:20.952648+0000","last_undegraded":"2022-03-10T01:29:20.952648+0000","last_fullsized":"2022-03-10T01:29:20.952648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-10T01:29:15.494246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-10T01:29:15.494246+0000","last_clean_scrub_stamp":"2022-03-10T01:29:15.494246+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-11T09:48:07.548181+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":4944,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740208,"statfs":{"total":95995035648,"available":95989972992,"internally_reserved":0,"allocated":139264,"data_stored":30866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5788,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739364,"statfs":{"total":95995035648,"available":95989108736,"internally_reserved":0,"allocated":610304,"data_stored":497822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5916,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739236,"statfs":{"total":95995035648,"available":95988977664,"internally_reserved":0,"allocated":610304,"data_stored":497822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5916,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739236,"statfs":{"total":95995035648,"available":95988977664,"internally_reserved":0,"allocated":610304,"data_stored":497822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-10T01:29:25.982 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-10T01:29:25.982 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-10T01:29:25.983 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-10T01:29:25.983 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph health --format=json 2022-03-10T01:29:26.521 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:26.521 INFO:teuthology.orchestra.run.smithi186.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-10T01:29:26.539 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-10T01:29:26.539 INFO:tasks.rook:Rook complete, yielding 2022-03-10T01:29:26.540 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-10T01:29:26.553 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph -s 2022-03-10T01:29:27.174 INFO:teuthology.orchestra.run.smithi186.stdout: cluster: 2022-03-10T01:29:27.175 INFO:teuthology.orchestra.run.smithi186.stdout: id: af064395-2045-4803-95db-56693404613d 2022-03-10T01:29:27.175 INFO:teuthology.orchestra.run.smithi186.stdout: health: HEALTH_OK 2022-03-10T01:29:27.175 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:27.175 INFO:teuthology.orchestra.run.smithi186.stdout: services: 2022-03-10T01:29:27.176 INFO:teuthology.orchestra.run.smithi186.stdout: mon: 1 daemons, quorum a (age 3m) 2022-03-10T01:29:27.176 INFO:teuthology.orchestra.run.smithi186.stdout: mgr: a(active, since 2m) 2022-03-10T01:29:27.176 INFO:teuthology.orchestra.run.smithi186.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 59s) 2022-03-10T01:29:27.176 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: data: 2022-03-10T01:29:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: pools: 1 pools, 1 pgs 2022-03-10T01:29:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: objects: 2 objects, 577 KiB 2022-03-10T01:29:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: usage: 22 MiB used, 358 GiB / 358 GiB avail 2022-03-10T01:29:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: pgs: 1 active+clean 2022-03-10T01:29:27.178 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:27.178 INFO:teuthology.orchestra.run.smithi186.stdout: io: 2022-03-10T01:29:27.178 INFO:teuthology.orchestra.run.smithi186.stdout: client: 1.5 KiB/s rd, 44 KiB/s wr, 1 op/s rd, 2 op/s wr 2022-03-10T01:29:27.178 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:29:27.193 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status 2022-03-10T01:29:27.751 INFO:teuthology.orchestra.run.smithi186.stdout:Backend: rook 2022-03-10T01:29:27.751 INFO:teuthology.orchestra.run.smithi186.stdout:Available: Yes 2022-03-10T01:29:27.775 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ps 2022-03-10T01:29:28.354 INFO:teuthology.orchestra.run.smithi186.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-10T01:29:28.355 INFO:teuthology.orchestra.run.smithi186.stdout:crashcollector.crash smithi186 running (60s) 0s ago 61s - - 4b871cde5e1a 2022-03-10T01:29:28.355 INFO:teuthology.orchestra.run.smithi186.stdout:mgr.a smithi186 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-10T01:29:28.355 INFO:teuthology.orchestra.run.smithi186.stdout:mon.a smithi186 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T01:29:28.355 INFO:teuthology.orchestra.run.smithi186.stdout:osd.0 smithi186 running (61s) 0s ago 62s - - 4b871cde5e1a 2022-03-10T01:29:28.355 INFO:teuthology.orchestra.run.smithi186.stdout:osd.1 smithi186 running (57s) 0s ago 58s - - 4b871cde5e1a 2022-03-10T01:29:28.356 INFO:teuthology.orchestra.run.smithi186.stdout:osd.2 smithi186 running (59s) 0s ago 60s - - 4b871cde5e1a 2022-03-10T01:29:28.356 INFO:teuthology.orchestra.run.smithi186.stdout:osd.3 smithi186 running (52s) 0s ago 53s - - 4b871cde5e1a 2022-03-10T01:29:28.376 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls 2022-03-10T01:29:28.990 INFO:teuthology.orchestra.run.smithi186.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-10T01:29:28.990 INFO:teuthology.orchestra.run.smithi186.stdout:crash 1/1 0s ago 62s * 2022-03-10T01:29:28.990 INFO:teuthology.orchestra.run.smithi186.stdout:mgr 1/1 0s ago 3m count:1 2022-03-10T01:29:28.991 INFO:teuthology.orchestra.run.smithi186.stdout:mon 1/1 0s ago 3m count:1 2022-03-10T01:29:28.991 INFO:teuthology.orchestra.run.smithi186.stdout:osd 8 0s ago 63s 2022-03-10T01:29:28.991 INFO:teuthology.orchestra.run.smithi186.stdout:osd.all-available-devices 0 0s ago - * 2022-03-10T01:29:29.014 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch device ls 2022-03-10T01:29:29.535 INFO:teuthology.orchestra.run.smithi186.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-10T01:29:29.535 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T01:29:29.535 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T01:29:29.536 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T01:29:29.536 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T01:29:29.556 INFO:teuthology.run_tasks:Running task install... 2022-03-10T01:29:29.567 DEBUG:teuthology.task.install:project ceph 2022-03-10T01:29:29.568 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3'}} 2022-03-10T01:29:29.568 DEBUG:teuthology.task.install:config {'host.a': None, 'flavor': 'default', 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3'} 2022-03-10T01:29:29.568 INFO:teuthology.task.install:Using flavor: default 2022-03-10T01:29:29.576 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-10T01:29:29.577 INFO:teuthology.task.install:extra packages: [] 2022-03-10T01:29:29.577 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-03-10T01:29:29.626 INFO:teuthology.orchestra.run.smithi186.stdout:Ubuntu 2022-03-10T01:29:29.632 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-10T01:29:29.632 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-key list | grep Ceph 2022-03-10T01:29:29.713 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-03-10T01:29:29.774 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-03-10T01:29:29.775 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph.com (release key) 2022-03-10T01:29:29.776 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-03-10T01:29:29.825 INFO:teuthology.orchestra.run.smithi186.stdout:Ubuntu 2022-03-10T01:29:29.830 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-10T01:29:29.831 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-03-10T01:29:29.832 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-10T01:29:29.989 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default/ 2022-03-10T01:29:29.990 INFO:teuthology.task.install.deb:Package version is 17.0.0-11025-g7a44e9eb-1focal 2022-03-10T01:29:30.121 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:29:30.122 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-03-10T01:29:30.136 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2022-03-10T01:29:30.314 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T01:29:30.394 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal InRelease 2022-03-10T01:29:30.417 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-03-10T01:29:30.437 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:29:30.442 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal Release.gpg 2022-03-10T01:29:30.465 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main i386 Packages [3,360 B] 2022-03-10T01:29:30.499 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2022-03-10T01:29:30.519 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T01:29:30.551 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease 2022-03-10T01:29:30.860 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T01:29:31.171 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 364 kB in 1s (455 kB/s) 2022-03-10T01:29:32.194 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:29:32.218 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-11025-g7a44e9eb-1focal cephadm=17.0.0-11025-g7a44e9eb-1focal ceph-mds=17.0.0-11025-g7a44e9eb-1focal ceph-mgr=17.0.0-11025-g7a44e9eb-1focal ceph-common=17.0.0-11025-g7a44e9eb-1focal ceph-fuse=17.0.0-11025-g7a44e9eb-1focal ceph-test=17.0.0-11025-g7a44e9eb-1focal ceph-volume=17.0.0-11025-g7a44e9eb-1focal radosgw=17.0.0-11025-g7a44e9eb-1focal python3-rados=17.0.0-11025-g7a44e9eb-1focal python3-rgw=17.0.0-11025-g7a44e9eb-1focal python3-cephfs=17.0.0-11025-g7a44e9eb-1focal python3-rbd=17.0.0-11025-g7a44e9eb-1focal libcephfs2=17.0.0-11025-g7a44e9eb-1focal libcephfs-dev=17.0.0-11025-g7a44e9eb-1focal librados2=17.0.0-11025-g7a44e9eb-1focal librbd1=17.0.0-11025-g7a44e9eb-1focal rbd-fuse=17.0.0-11025-g7a44e9eb-1focal 2022-03-10T01:29:32.294 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2022-03-10T01:29:32.476 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2022-03-10T01:29:32.477 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2022-03-10T01:29:32.641 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages were automatically installed and are no longer required: 2022-03-10T01:29:32.642 INFO:teuthology.orchestra.run.smithi186.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-03-10T01:29:32.643 INFO:teuthology.orchestra.run.smithi186.stdout:Use 'sudo apt autoremove' to remove them. 2022-03-10T01:29:32.643 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2022-03-10T01:29:32.643 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-03-10T01:29:32.645 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjq1 2022-03-10T01:29:32.645 INFO:teuthology.orchestra.run.smithi186.stdout: libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-10T01:29:32.645 INFO:teuthology.orchestra.run.smithi186.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-10T01:29:32.646 INFO:teuthology.orchestra.run.smithi186.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-10T01:29:32.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-10T01:29:32.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-03-10T01:29:32.646 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-10T01:29:32.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-10T01:29:32.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-10T01:29:32.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-10T01:29:32.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-10T01:29:32.647 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-10T01:29:32.648 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-03-10T01:29:32.648 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-03-10T01:29:32.648 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-03-10T01:29:32.648 INFO:teuthology.orchestra.run.smithi186.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-03-10T01:29:32.649 INFO:teuthology.orchestra.run.smithi186.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-03-10T01:29:32.649 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket python3-webtest python3-werkzeug xmlstarlet 2022-03-10T01:29:32.649 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2022-03-10T01:29:32.649 INFO:teuthology.orchestra.run.smithi186.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-03-10T01:29:32.649 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-vectors python3-genshi python3-crypto python3-lxml-dbg 2022-03-10T01:29:32.650 INFO:teuthology.orchestra.run.smithi186.stdout: python-lxml-doc python3-beaker python-mako-doc python-openssl-doc 2022-03-10T01:29:32.650 INFO:teuthology.orchestra.run.smithi186.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2022-03-10T01:29:32.650 INFO:teuthology.orchestra.run.smithi186.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2022-03-10T01:29:32.650 INFO:teuthology.orchestra.run.smithi186.stdout: python-pyinotify-doc python-scipy-doc python3-dap python-sklearn-doc 2022-03-10T01:29:32.651 INFO:teuthology.orchestra.run.smithi186.stdout: ipython3 python-waitress-doc python-webob-doc python-webtest-doc 2022-03-10T01:29:32.651 INFO:teuthology.orchestra.run.smithi186.stdout: python-werkzeug-doc python3-termcolor python3-watchdog 2022-03-10T01:29:32.651 INFO:teuthology.orchestra.run.smithi186.stdout:Recommended packages: 2022-03-10T01:29:32.651 INFO:teuthology.orchestra.run.smithi186.stdout: btrfs-tools ceph-mgr-cephadm 2022-03-10T01:29:32.742 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2022-03-10T01:29:32.742 INFO:teuthology.orchestra.run.smithi186.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-dashboard 2022-03-10T01:29:32.742 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-03-10T01:29:32.744 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon ceph-osd ceph-test ceph-volume cephadm jq libcephfs-dev libcephfs2 2022-03-10T01:29:32.745 INFO:teuthology.orchestra.run.smithi186.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-10T01:29:32.745 INFO:teuthology.orchestra.run.smithi186.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-10T01:29:32.746 INFO:teuthology.orchestra.run.smithi186.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-10T01:29:32.746 INFO:teuthology.orchestra.run.smithi186.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-10T01:29:32.746 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2022-03-10T01:29:32.746 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-10T01:29:32.747 INFO:teuthology.orchestra.run.smithi186.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-10T01:29:32.747 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-10T01:29:32.747 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-10T01:29:32.747 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-10T01:29:32.747 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-10T01:29:32.748 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2022-03-10T01:29:32.748 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2022-03-10T01:29:32.748 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-03-10T01:29:32.748 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-03-10T01:29:32.749 INFO:teuthology.orchestra.run.smithi186.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-03-10T01:29:32.749 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-03-10T01:29:32.749 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse xmlstarlet 2022-03-10T01:29:32.749 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages will be upgraded: 2022-03-10T01:29:32.749 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 librbd1 2022-03-10T01:29:32.904 INFO:teuthology.orchestra.run.smithi186.stdout:2 upgraded, 95 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T01:29:32.905 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 181 MB of archives. 2022-03-10T01:29:32.905 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 845 MB of additional disk space will be used. 2022-03-10T01:29:32.905 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-11025-g7a44e9eb-1focal [3,484 kB] 2022-03-10T01:29:32.937 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-03-10T01:29:33.313 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-03-10T01:29:33.314 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-03-10T01:29:33.319 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-03-10T01:29:33.328 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-03-10T01:29:33.356 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-11025-g7a44e9eb-1focal [3,533 kB] 2022-03-10T01:29:33.491 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-03-10T01:29:33.498 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-03-10T01:29:33.533 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-03-10T01:29:33.539 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-03-10T01:29:33.541 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-03-10T01:29:33.561 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-03-10T01:29:33.563 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-03-10T01:29:33.564 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-03-10T01:29:33.565 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-11025-g7a44e9eb-1focal [755 kB] 2022-03-10T01:29:33.566 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-03-10T01:29:33.573 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-03-10T01:29:33.604 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-11025-g7a44e9eb-1focal [412 kB] 2022-03-10T01:29:33.615 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-11025-g7a44e9eb-1focal [22.4 kB] 2022-03-10T01:29:33.615 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-11025-g7a44e9eb-1focal [200 kB] 2022-03-10T01:29:33.629 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-11025-g7a44e9eb-1focal [41.9 kB] 2022-03-10T01:29:33.630 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-11025-g7a44e9eb-1focal [385 kB] 2022-03-10T01:29:33.648 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-11025-g7a44e9eb-1focal [4,162 kB] 2022-03-10T01:29:33.653 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-03-10T01:29:33.653 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-03-10T01:29:33.657 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-03-10T01:29:33.658 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-03-10T01:29:33.661 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2022-03-10T01:29:33.664 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-03-10T01:29:33.664 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-03-10T01:29:33.678 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-03-10T01:29:33.678 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-03-10T01:29:33.739 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-03-10T01:29:33.739 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-03-10T01:29:33.798 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-11025-g7a44e9eb-1focal [116 kB] 2022-03-10T01:29:33.821 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-03-10T01:29:33.821 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-11025-g7a44e9eb-1focal [478 kB] 2022-03-10T01:29:33.825 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-11025-g7a44e9eb-1focal [24.4 MB] 2022-03-10T01:29:33.827 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-03-10T01:29:33.828 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-03-10T01:29:33.832 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-03-10T01:29:33.835 INFO:teuthology.orchestra.run.smithi186.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-03-10T01:29:33.839 INFO:teuthology.orchestra.run.smithi186.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-03-10T01:29:33.840 INFO:teuthology.orchestra.run.smithi186.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-03-10T01:29:33.840 INFO:teuthology.orchestra.run.smithi186.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-03-10T01:29:34.196 INFO:teuthology.orchestra.run.smithi186.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-03-10T01:29:34.248 INFO:teuthology.orchestra.run.smithi186.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-03-10T01:29:34.250 INFO:teuthology.orchestra.run.smithi186.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-03-10T01:29:34.298 INFO:teuthology.orchestra.run.smithi186.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-03-10T01:29:34.299 INFO:teuthology.orchestra.run.smithi186.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-03-10T01:29:34.299 INFO:teuthology.orchestra.run.smithi186.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-03-10T01:29:34.316 INFO:teuthology.orchestra.run.smithi186.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-03-10T01:29:34.318 INFO:teuthology.orchestra.run.smithi186.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-03-10T01:29:34.319 INFO:teuthology.orchestra.run.smithi186.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-03-10T01:29:34.320 INFO:teuthology.orchestra.run.smithi186.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-03-10T01:29:34.362 INFO:teuthology.orchestra.run.smithi186.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-03-10T01:29:34.363 INFO:teuthology.orchestra.run.smithi186.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-03-10T01:29:34.444 INFO:teuthology.orchestra.run.smithi186.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-03-10T01:29:34.448 INFO:teuthology.orchestra.run.smithi186.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-03-10T01:29:34.451 INFO:teuthology.orchestra.run.smithi186.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-03-10T01:29:34.453 INFO:teuthology.orchestra.run.smithi186.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-03-10T01:29:34.454 INFO:teuthology.orchestra.run.smithi186.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-03-10T01:29:34.456 INFO:teuthology.orchestra.run.smithi186.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-03-10T01:29:34.458 INFO:teuthology.orchestra.run.smithi186.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-03-10T01:29:34.461 INFO:teuthology.orchestra.run.smithi186.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-03-10T01:29:34.526 INFO:teuthology.orchestra.run.smithi186.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-03-10T01:29:34.527 INFO:teuthology.orchestra.run.smithi186.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-03-10T01:29:34.607 INFO:teuthology.orchestra.run.smithi186.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-03-10T01:29:34.612 INFO:teuthology.orchestra.run.smithi186.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-03-10T01:29:34.630 INFO:teuthology.orchestra.run.smithi186.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-03-10T01:29:34.631 INFO:teuthology.orchestra.run.smithi186.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-03-10T01:29:34.631 INFO:teuthology.orchestra.run.smithi186.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-03-10T01:29:34.632 INFO:teuthology.orchestra.run.smithi186.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-03-10T01:29:34.633 INFO:teuthology.orchestra.run.smithi186.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-03-10T01:29:34.634 INFO:teuthology.orchestra.run.smithi186.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-03-10T01:29:34.692 INFO:teuthology.orchestra.run.smithi186.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-03-10T01:29:34.749 INFO:teuthology.orchestra.run.smithi186.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-03-10T01:29:34.749 INFO:teuthology.orchestra.run.smithi186.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-11025-g7a44e9eb-1focal [6,433 kB] 2022-03-10T01:29:34.775 INFO:teuthology.orchestra.run.smithi186.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-03-10T01:29:34.846 INFO:teuthology.orchestra.run.smithi186.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-11025-g7a44e9eb-1focal [211 kB] 2022-03-10T01:29:34.851 INFO:teuthology.orchestra.run.smithi186.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-11025-g7a44e9eb-1focal [151 kB] 2022-03-10T01:29:34.856 INFO:teuthology.orchestra.run.smithi186.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-11025-g7a44e9eb-1focal [1,574 kB] 2022-03-10T01:29:34.907 INFO:teuthology.orchestra.run.smithi186.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-11025-g7a44e9eb-1focal [7,824 kB] 2022-03-10T01:29:35.176 INFO:teuthology.orchestra.run.smithi186.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-11025-g7a44e9eb-1focal [28.6 MB] 2022-03-10T01:29:36.217 INFO:teuthology.orchestra.run.smithi186.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-11025-g7a44e9eb-1focal [3,808 B] 2022-03-10T01:29:36.217 INFO:teuthology.orchestra.run.smithi186.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-11025-g7a44e9eb-1focal [863 kB] 2022-03-10T01:29:36.223 INFO:teuthology.orchestra.run.smithi186.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-11025-g7a44e9eb-1focal [2,396 kB] 2022-03-10T01:29:36.242 INFO:teuthology.orchestra.run.smithi186.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-11025-g7a44e9eb-1focal [1,677 kB] 2022-03-10T01:29:36.274 INFO:teuthology.orchestra.run.smithi186.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-11025-g7a44e9eb-1focal [7,889 kB] 2022-03-10T01:29:36.534 INFO:teuthology.orchestra.run.smithi186.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-11025-g7a44e9eb-1focal [3,924 B] 2022-03-10T01:29:36.534 INFO:teuthology.orchestra.run.smithi186.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-11025-g7a44e9eb-1focal [50.5 MB] 2022-03-10T01:29:38.501 INFO:teuthology.orchestra.run.smithi186.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-11025-g7a44e9eb-1focal [219 kB] 2022-03-10T01:29:38.503 INFO:teuthology.orchestra.run.smithi186.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-11025-g7a44e9eb-1focal [69.5 kB] 2022-03-10T01:29:38.504 INFO:teuthology.orchestra.run.smithi186.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-11025-g7a44e9eb-1focal [21.7 kB] 2022-03-10T01:29:38.504 INFO:teuthology.orchestra.run.smithi186.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-11025-g7a44e9eb-1focal [12.7 MB] 2022-03-10T01:29:38.809 INFO:teuthology.orchestra.run.smithi186.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-03-07-0958/7a44e9ebca556862e9704be295a07719d7db47b3/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-11025-g7a44e9eb-1focal [78.8 kB] 2022-03-10T01:29:39.408 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 181 MB in 6s (29.9 MB/s) 2022-03-10T01:29:39.585 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-03-10T01:29:39.631 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109337 files and directories currently installed.) 2022-03-10T01:29:39.634 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../00-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-03-10T01:29:39.725 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-10T01:29:40.131 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../01-librbd1_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:40.296 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librbd1 (17.0.0-11025-g7a44e9eb-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-10T01:29:41.076 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../02-librados2_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:41.241 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librados2 (17.0.0-11025-g7a44e9eb-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-10T01:29:42.089 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs2. 2022-03-10T01:29:42.102 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../03-libcephfs2_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:42.138 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:43.149 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rados. 2022-03-10T01:29:43.165 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../04-python3-rados_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:43.244 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rados (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:43.641 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-03-10T01:29:43.651 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../05-python3-ceph-argparse_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-10T01:29:43.691 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-argparse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:43.982 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cephfs. 2022-03-10T01:29:43.995 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../06-python3-cephfs_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:44.032 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cephfs (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:44.590 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-common. 2022-03-10T01:29:44.603 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../07-python3-ceph-common_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-10T01:29:44.640 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:45.014 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-prettytable. 2022-03-10T01:29:45.025 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../08-python3-prettytable_0.7.2-5_all.deb ... 2022-03-10T01:29:45.072 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-03-10T01:29:45.690 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rbd. 2022-03-10T01:29:45.703 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../09-python3-rbd_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:45.740 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rbd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:46.149 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-03-10T01:29:46.161 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../10-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-03-10T01:29:46.219 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-10T01:29:46.850 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-03-10T01:29:46.865 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../11-librabbitmq4_0.10.0-1_amd64.deb ... 2022-03-10T01:29:46.908 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-03-10T01:29:47.460 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-03-10T01:29:47.472 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../12-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-03-10T01:29:47.520 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-10T01:29:48.086 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librgw2. 2022-03-10T01:29:48.098 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../13-librgw2_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:48.137 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librgw2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:48.925 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rgw. 2022-03-10T01:29:48.938 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../14-python3-rgw_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:49.047 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:49.550 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liboath0. 2022-03-10T01:29:49.559 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../15-liboath0_2.6.1-1.3_amd64.deb ... 2022-03-10T01:29:49.625 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-03-10T01:29:49.987 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libradosstriper1. 2022-03-10T01:29:49.996 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../16-libradosstriper1_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:50.059 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libradosstriper1 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:50.611 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-common. 2022-03-10T01:29:50.624 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../17-ceph-common_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:50.678 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:53.595 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-base. 2022-03-10T01:29:53.609 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../18-ceph-base_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:29:53.665 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-base (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:54.912 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cryptography. 2022-03-10T01:29:54.925 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../19-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-03-10T01:29:54.962 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-10T01:29:55.735 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-openssl. 2022-03-10T01:29:55.748 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../20-python3-openssl_19.0.0-1build1_all.deb ... 2022-03-10T01:29:55.826 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-03-10T01:29:56.212 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-03-10T01:29:56.228 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../21-ceph-mgr-modules-core_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-10T01:29:56.270 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-modules-core (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:29:57.030 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-bcrypt. 2022-03-10T01:29:57.034 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../22-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-03-10T01:29:57.120 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-10T01:29:57.540 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cherrypy3. 2022-03-10T01:29:57.557 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../23-python3-cherrypy3_8.9.1-6_all.deb ... 2022-03-10T01:29:57.599 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-03-10T01:29:57.998 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jwt. 2022-03-10T01:29:58.012 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../24-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-03-10T01:29:58.106 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-03-10T01:29:58.459 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-logutils. 2022-03-10T01:29:58.468 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../25-python3-logutils_0.3.3-7_all.deb ... 2022-03-10T01:29:58.509 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-03-10T01:29:59.001 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-markupsafe. 2022-03-10T01:29:59.013 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../26-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-03-10T01:29:59.057 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-03-10T01:29:59.505 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-mako. 2022-03-10T01:29:59.517 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../27-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-03-10T01:29:59.572 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-10T01:30:00.027 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplegeneric. 2022-03-10T01:30:00.040 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../28-python3-simplegeneric_0.8.1-3_all.deb ... 2022-03-10T01:30:00.085 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-03-10T01:30:00.450 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-singledispatch. 2022-03-10T01:30:00.463 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../29-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-03-10T01:30:00.515 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-03-10T01:30:00.857 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webob. 2022-03-10T01:30:00.871 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../30-python3-webob_1%3a1.8.5-2_all.deb ... 2022-03-10T01:30:00.945 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-03-10T01:30:01.353 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-soupsieve. 2022-03-10T01:30:01.366 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../31-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-03-10T01:30:01.421 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-10T01:30:02.036 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-bs4. 2022-03-10T01:30:02.049 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../32-python3-bs4_4.8.2-1_all.deb ... 2022-03-10T01:30:02.104 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-03-10T01:30:02.565 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-waitress. 2022-03-10T01:30:02.575 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../33-python3-waitress_1.4.1-1_all.deb ... 2022-03-10T01:30:02.618 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-waitress (1.4.1-1) ... 2022-03-10T01:30:03.102 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-tempita. 2022-03-10T01:30:03.115 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../34-python3-tempita_0.5.2-6_all.deb ... 2022-03-10T01:30:03.260 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-03-10T01:30:04.655 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-paste. 2022-03-10T01:30:04.667 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../35-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-03-10T01:30:04.705 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-03-10T01:30:05.107 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-03-10T01:30:05.121 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-03-10T01:30:05.224 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-03-10T01:30:05.808 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastedeploy. 2022-03-10T01:30:05.822 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../37-python3-pastedeploy_2.1.0-1_all.deb ... 2022-03-10T01:30:05.870 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-03-10T01:30:06.248 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webtest. 2022-03-10T01:30:06.261 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../38-python3-webtest_2.0.32-2_all.deb ... 2022-03-10T01:30:06.316 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-03-10T01:30:06.935 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pecan. 2022-03-10T01:30:06.952 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../39-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-03-10T01:30:07.031 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-03-10T01:30:07.560 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-werkzeug. 2022-03-10T01:30:07.569 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../40-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-03-10T01:30:07.610 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-10T01:30:08.013 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-03-10T01:30:08.026 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../41-libsqlite3-mod-ceph_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:08.104 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:08.599 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr. 2022-03-10T01:30:08.613 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../42-ceph-mgr_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:08.669 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:09.911 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mon. 2022-03-10T01:30:09.921 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../43-ceph-mon_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:09.961 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mon (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:11.362 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-osd. 2022-03-10T01:30:11.377 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../44-ceph-osd_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:11.444 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-osd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:14.612 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph. 2022-03-10T01:30:14.624 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../45-ceph_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:14.669 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:15.214 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-fuse. 2022-03-10T01:30:15.229 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../46-ceph-fuse_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:15.306 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:15.850 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mds. 2022-03-10T01:30:15.865 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../47-ceph-mds_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:15.903 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mds (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:16.511 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-repoze.lru. 2022-03-10T01:30:16.525 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2022-03-10T01:30:16.599 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-03-10T01:30:17.140 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjs-underscore. 2022-03-10T01:30:17.152 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../49-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-03-10T01:30:17.189 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-10T01:30:17.550 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-03-10T01:30:17.563 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../50-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-03-10T01:30:17.650 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-10T01:30:18.013 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-routes. 2022-03-10T01:30:18.025 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../51-python3-routes_2.4.1-2_all.deb ... 2022-03-10T01:30:18.065 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-03-10T01:30:18.501 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-03-10T01:30:18.514 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../52-ceph-mgr-dashboard_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-10T01:30:18.587 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-dashboard (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:19.271 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-decorator. 2022-03-10T01:30:19.286 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../53-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-03-10T01:30:19.371 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-03-10T01:30:19.746 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-03-10T01:30:19.760 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../54-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-03-10T01:30:19.804 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-10T01:30:20.294 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-scipy. 2022-03-10T01:30:20.303 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../55-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-03-10T01:30:20.346 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-03-10T01:30:22.294 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-03-10T01:30:22.306 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../56-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-03-10T01:30:22.356 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-10T01:30:23.176 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-joblib. 2022-03-10T01:30:23.189 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../57-python3-joblib_0.14.0-3_all.deb ... 2022-03-10T01:30:23.233 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-03-10T01:30:23.633 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn. 2022-03-10T01:30:23.646 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../58-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-03-10T01:30:23.691 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-10T01:30:24.683 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-03-10T01:30:24.696 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../59-ceph-mgr-diskprediction-local_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-10T01:30:24.783 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:26.188 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cachetools. 2022-03-10T01:30:26.200 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../60-python3-cachetools_4.0.0-1_all.deb ... 2022-03-10T01:30:26.237 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-03-10T01:30:26.643 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1. 2022-03-10T01:30:26.651 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../61-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-03-10T01:30:26.690 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-03-10T01:30:27.300 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-03-10T01:30:27.311 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../62-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-03-10T01:30:27.386 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-10T01:30:27.804 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rsa. 2022-03-10T01:30:27.818 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../63-python3-rsa_4.0-3_all.deb ... 2022-03-10T01:30:27.862 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rsa (4.0-3) ... 2022-03-10T01:30:28.261 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-google-auth. 2022-03-10T01:30:28.273 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../64-python3-google-auth_1.5.1-2_all.deb ... 2022-03-10T01:30:28.327 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-03-10T01:30:28.790 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-blinker. 2022-03-10T01:30:28.805 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../65-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-03-10T01:30:28.864 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-10T01:30:29.227 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-oauthlib. 2022-03-10T01:30:29.239 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../66-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-03-10T01:30:29.276 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-10T01:30:29.718 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-03-10T01:30:29.728 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../67-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-03-10T01:30:29.776 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-10T01:30:30.388 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-websocket. 2022-03-10T01:30:30.399 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../68-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-03-10T01:30:30.497 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-03-10T01:30:30.826 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-kubernetes. 2022-03-10T01:30:30.839 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../69-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-03-10T01:30:30.879 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-03-10T01:30:31.582 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-03-10T01:30:31.596 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../70-ceph-mgr-k8sevents_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-10T01:30:31.711 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:32.161 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libonig5:amd64. 2022-03-10T01:30:32.173 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../71-libonig5_6.9.4-1_amd64.deb ... 2022-03-10T01:30:32.244 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-03-10T01:30:32.823 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjq1:amd64. 2022-03-10T01:30:32.832 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../72-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-10T01:30:32.932 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-10T01:30:33.253 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package jq. 2022-03-10T01:30:33.267 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../73-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-10T01:30:33.314 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-03-10T01:30:33.700 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-03-10T01:30:33.709 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../74-libxslt1.1_1.1.34-4_amd64.deb ... 2022-03-10T01:30:33.767 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-03-10T01:30:34.205 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package xmlstarlet. 2022-03-10T01:30:34.213 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../75-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-03-10T01:30:34.254 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-03-10T01:30:34.714 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-test. 2022-03-10T01:30:34.726 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../76-ceph-test_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:34.822 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-test (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:39.663 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-volume. 2022-03-10T01:30:39.674 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../77-ceph-volume_17.0.0-11025-g7a44e9eb-1focal_all.deb ... 2022-03-10T01:30:39.766 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-volume (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:40.256 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package cephadm. 2022-03-10T01:30:40.272 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../78-cephadm_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:40.351 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking cephadm (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:41.332 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs-dev. 2022-03-10T01:30:41.349 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../79-libcephfs-dev_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:41.474 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs-dev (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:42.162 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package nvme-cli. 2022-03-10T01:30:42.177 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../80-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-03-10T01:30:42.235 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-03-10T01:30:43.023 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-atomicwrites. 2022-03-10T01:30:43.036 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../81-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-03-10T01:30:43.084 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-03-10T01:30:43.428 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-attr. 2022-03-10T01:30:43.442 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../82-python3-attr_19.3.0-2_all.deb ... 2022-03-10T01:30:43.494 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-03-10T01:30:43.839 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webencodings. 2022-03-10T01:30:43.853 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../83-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-03-10T01:30:43.931 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-10T01:30:44.425 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-html5lib. 2022-03-10T01:30:44.438 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../84-python3-html5lib_1.0.1-2_all.deb ... 2022-03-10T01:30:44.500 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-03-10T01:30:44.954 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-03-10T01:30:44.967 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../85-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-03-10T01:30:45.008 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-10T01:30:45.697 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-packaging. 2022-03-10T01:30:45.711 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../86-python3-packaging_20.3-1_all.deb ... 2022-03-10T01:30:45.755 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-packaging (20.3-1) ... 2022-03-10T01:30:46.226 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastescript. 2022-03-10T01:30:46.240 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../87-python3-pastescript_2.0.2-4_all.deb ... 2022-03-10T01:30:46.289 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-03-10T01:30:46.938 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pluggy. 2022-03-10T01:30:46.950 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../88-python3-pluggy_0.13.0-2_all.deb ... 2022-03-10T01:30:46.996 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-03-10T01:30:47.561 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-psutil. 2022-03-10T01:30:47.573 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../89-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-03-10T01:30:47.655 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-03-10T01:30:48.141 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-py. 2022-03-10T01:30:48.149 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../90-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-03-10T01:30:48.211 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-03-10T01:30:48.655 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyinotify. 2022-03-10T01:30:48.667 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../91-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-03-10T01:30:48.759 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-10T01:30:49.175 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-wcwidth. 2022-03-10T01:30:49.187 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../92-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-03-10T01:30:49.224 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-10T01:30:49.560 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pytest. 2022-03-10T01:30:49.573 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../93-python3-pytest_4.6.9-1_all.deb ... 2022-03-10T01:30:49.609 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-03-10T01:30:50.173 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplejson. 2022-03-10T01:30:50.188 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../94-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-03-10T01:30:50.231 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-10T01:30:50.849 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package radosgw. 2022-03-10T01:30:50.862 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../95-radosgw_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:50.940 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking radosgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:52.543 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package rbd-fuse. 2022-03-10T01:30:52.555 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../96-rbd-fuse_17.0.0-11025-g7a44e9eb-1focal_amd64.deb ... 2022-03-10T01:30:52.592 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking rbd-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:53.103 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-attr (19.3.0-2) ... 2022-03-10T01:30:53.466 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-03-10T01:30:53.599 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-03-10T01:30:53.843 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up cephadm (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:54.762 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user cephadm....done 2022-03-10T01:30:54.967 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-waitress (1.4.1-1) ... 2022-03-10T01:30:55.335 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-03-10T01:30:55.644 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-03-10T01:30:55.769 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-03-10T01:30:56.095 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-03-10T01:30:56.596 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-03-10T01:30:56.940 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-03-10T01:30:57.275 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-10T01:30:57.718 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-10T01:30:58.042 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-argparse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:30:58.394 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-10T01:30:58.595 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-03-10T01:30:58.921 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-10T01:30:59.230 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-03-10T01:30:59.631 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-03-10T01:30:59.948 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-10T01:31:00.377 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-03-10T01:31:00.742 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-03-10T01:31:01.128 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-packaging (20.3-1) ... 2022-03-10T01:31:01.427 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-03-10T01:31:01.561 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-10T01:31:01.995 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-03-10T01:31:02.490 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-10T01:31:02.950 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-10T01:31:03.410 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:03.712 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-03-10T01:31:04.054 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-10T01:31:04.272 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-03-10T01:31:04.662 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-10T01:31:04.873 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-03-10T01:31:05.007 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-03-10T01:31:05.383 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-03-10T01:31:05.776 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-03-10T01:31:06.124 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-10T01:31:06.395 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-03-10T01:31:06.695 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-03-10T01:31:07.156 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-10T01:31:07.516 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-03-10T01:31:07.738 INFO:teuthology.orchestra.run.smithi186.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-03-10T01:31:07.989 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-03-10T01:31:08.141 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-10T01:31:08.451 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-03-10T01:31:08.869 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-10T01:31:09.028 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-10T01:31:09.154 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-03-10T01:31:09.495 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-10T01:31:09.780 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librados2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:09.931 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-03-10T01:31:11.638 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librgw2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:11.799 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libsqlite3-mod-ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:11.941 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-10T01:31:12.158 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-03-10T01:31:13.318 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-03-10T01:31:13.653 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-03-10T01:31:13.962 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-03-10T01:31:14.338 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs2 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:14.465 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-10T01:31:14.830 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libradosstriper1 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:15.049 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-10T01:31:15.416 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-10T01:31:16.655 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librbd1 (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:16.830 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-10T01:31:16.981 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-03-10T01:31:17.174 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-10T01:31:17.615 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-03-10T01:31:17.811 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rsa (4.0-3) ... 2022-03-10T01:31:18.171 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-03-10T01:31:18.546 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-modules-core (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:18.754 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:19.095 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-10T01:31:19.095 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-10T01:31:20.071 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs-dev (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:20.242 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-10T01:31:20.593 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-03-10T01:31:21.023 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-03-10T01:31:21.439 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rados (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:21.588 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rbd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:21.789 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up rbd-fuse (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:21.982 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-routes (2.4.1-2) ... 2022-03-10T01:31:22.326 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-03-10T01:31:22.634 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:22.850 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cephfs (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:23.117 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-03-10T01:31:24.397 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-common (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:25.205 INFO:teuthology.orchestra.run.smithi186.stdout:Adding group ceph....done 2022-03-10T01:31:25.748 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user ceph....done 2022-03-10T01:31:25.994 INFO:teuthology.orchestra.run.smithi186.stdout:Setting system user ceph properties....done 2022-03-10T01:31:26.013 INFO:teuthology.orchestra.run.smithi186.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-03-10T01:31:26.165 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-03-10T01:31:26.769 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-03-10T01:31:27.817 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-test (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:27.979 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up radosgw (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:28.299 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-10T01:31:28.300 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-10T01:31:29.657 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-base (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:30.068 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-03-10T01:31:31.039 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mds (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:31.306 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-10T01:31:31.306 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-10T01:31:32.316 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:32.562 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-10T01:31:32.563 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-10T01:31:33.604 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-osd (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:34.523 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-10T01:31:34.523 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-10T01:31:35.503 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-k8sevents (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:35.707 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:35.866 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mon (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:36.084 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-10T01:31:36.085 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-10T01:31:37.107 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:37.305 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-dashboard (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:37.421 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-volume (17.0.0-11025-g7a44e9eb-1focal) ... 2022-03-10T01:31:37.710 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for systemd (245.4-4ubuntu3.15) ... 2022-03-10T01:31:38.328 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T01:31:40.988 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T01:31:41.481 INFO:teuthology.orchestra.run.smithi186.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-03-10T01:31:41.482 DEBUG:teuthology.parallel:result is None 2022-03-10T01:31:41.483 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-03-10T01:31:41.529 INFO:teuthology.orchestra.run.smithi186.stdout:Ubuntu 2022-03-10T01:31:41.534 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-10T01:31:41.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-10T01:31:41.718 DEBUG:teuthology.orchestra.run.smithi186:> dpkg-query -W -f '${Version}' ceph 2022-03-10T01:31:41.739 INFO:teuthology.orchestra.run.smithi186.stdout:17.0.0-11025-g7a44e9eb-1focal 2022-03-10T01:31:41.739 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11025-g7a44e9eb-1focal 2022-03-10T01:31:41.740 INFO:teuthology.task.install:The correct ceph version 17.0.0-11025-g7a44e9eb-1focal is installed. 2022-03-10T01:31:41.740 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-10T01:31:41.741 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:31:41.741 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-10T01:31:41.753 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-10T01:31:41.754 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:31:41.755 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2022-03-10T01:31:41.812 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-10T01:31:41.822 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-10T01:31:41.823 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-03-10T01:31:41.823 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-10T01:31:41.834 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-10T01:31:41.892 INFO:teuthology.run_tasks:Running task radosbench... 2022-03-10T01:31:41.912 INFO:tasks.radosbench:Beginning radosbench... 2022-03-10T01:31:41.913 INFO:tasks.rook.ceph_manager.ceph:creating pool_name unique_pool_0 2022-03-10T01:31:41.914 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd pool create unique_pool_0 16 2022-03-10T01:31:42.981 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'unique_pool_0' created 2022-03-10T01:31:42.997 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2022-03-10T01:31:44.046 INFO:teuthology.orchestra.run.smithi186.stderr:enabled application 'rados' on pool 'unique_pool_0' 2022-03-10T01:31:45.075 DEBUG:teuthology.orchestra.run.smithi186:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.a -b 65536 --object-size 65536 -p unique_pool_0 bench 360 write' 2022-03-10T01:31:45.078 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-10T01:31:45.090 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch host label add '`hostname`' 'foo 2022-03-10T01:31:45.091 DEBUG:teuthology.orchestra.run.smithi186:> ceph' orch host ls '|' grep 'foo 2022-03-10T01:31:45.091 DEBUG:teuthology.orchestra.run.smithi186:> ceph' orch host label rm '`hostname`' 'foo 2022-03-10T01:31:45.091 DEBUG:teuthology.orchestra.run.smithi186:> ceph' orch host ls '|' grep -v 'foo 2022-03-10T01:31:45.092 DEBUG:teuthology.orchestra.run.smithi186:> ' 2022-03-10T01:31:45.123 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:hints = 1 2022-03-10T01:31:45.123 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 360 seconds or 0 objects 2022-03-10T01:31:45.124 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Object prefix: benchmark_data_smithi186_50823 2022-03-10T01:31:45.124 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:31:45.124 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 0 0 0 0 0 0 - 0 2022-03-10T01:31:45.462 INFO:teuthology.orchestra.run.smithi186.stdout:ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-03-10T01:31:45.488 INFO:teuthology.run_tasks:Running task exec... 2022-03-10T01:31:45.502 INFO:teuthology.task.exec:Executing custom commands... 2022-03-10T01:31:45.502 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi186.front.sepia.ceph.com 2022-03-10T01:31:45.503 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-10T01:31:45.503 DEBUG:teuthology.orchestra.run.smithi186:> toolbox() { 2022-03-10T01:31:45.503 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-10T01:31:45.503 DEBUG:teuthology.orchestra.run.smithi186:> } 2022-03-10T01:31:45.504 DEBUG:teuthology.orchestra.run.smithi186:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-10T01:31:45.504 DEBUG:teuthology.orchestra.run.smithi186:> toolbox ceph orch osd rm 0 --force 2022-03-10T01:31:45.504 DEBUG:teuthology.orchestra.run.smithi186:> removed_pv="" 2022-03-10T01:31:45.505 DEBUG:teuthology.orchestra.run.smithi186:> while [ "$removed_pv" = "" ] 2022-03-10T01:31:45.505 DEBUG:teuthology.orchestra.run.smithi186:> do 2022-03-10T01:31:45.506 DEBUG:teuthology.orchestra.run.smithi186:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-10T01:31:45.506 DEBUG:teuthology.orchestra.run.smithi186:> sleep 3s 2022-03-10T01:31:45.506 DEBUG:teuthology.orchestra.run.smithi186:> done 2022-03-10T01:31:45.506 DEBUG:teuthology.orchestra.run.smithi186:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-10T01:31:45.507 DEBUG:teuthology.orchestra.run.smithi186:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-10T01:31:45.507 DEBUG:teuthology.orchestra.run.smithi186:> toolbox ceph orch device zap $host $target_path --force 2022-03-10T01:31:45.507 DEBUG:teuthology.orchestra.run.smithi186:> zap_completion="0" 2022-03-10T01:31:45.508 DEBUG:teuthology.orchestra.run.smithi186:> while [ "$zap_completion" = "0" ] 2022-03-10T01:31:45.508 DEBUG:teuthology.orchestra.run.smithi186:> do 2022-03-10T01:31:45.508 DEBUG:teuthology.orchestra.run.smithi186:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-10T01:31:45.508 DEBUG:teuthology.orchestra.run.smithi186:> sleep 3s 2022-03-10T01:31:45.509 DEBUG:teuthology.orchestra.run.smithi186:> done 2022-03-10T01:31:45.509 DEBUG:teuthology.orchestra.run.smithi186:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-10T01:31:45.509 DEBUG:teuthology.orchestra.run.smithi186:> toolbox ceph orch apply osd --all-available-devices 2022-03-10T01:31:45.510 DEBUG:teuthology.orchestra.run.smithi186:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-10T01:31:45.510 DEBUG:teuthology.orchestra.run.smithi186:> num_osd="0" 2022-03-10T01:31:45.510 DEBUG:teuthology.orchestra.run.smithi186:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-10T01:31:45.511 DEBUG:teuthology.orchestra.run.smithi186:> do 2022-03-10T01:31:45.511 DEBUG:teuthology.orchestra.run.smithi186:> echo "waiting for osd to come back up" 2022-03-10T01:31:45.511 DEBUG:teuthology.orchestra.run.smithi186:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-10T01:31:45.511 DEBUG:teuthology.orchestra.run.smithi186:> sleep 30s 2022-03-10T01:31:45.512 DEBUG:teuthology.orchestra.run.smithi186:> done 2022-03-10T01:31:45.512 DEBUG:teuthology.orchestra.run.smithi186:> ' 2022-03-10T01:31:45.525 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-10T01:31:45.525 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:31:45.525 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-10T01:31:46.124 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 1 16 1274 1258 78.6207 78.625 0.0077898 0.01263 2022-03-10T01:31:47.124 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 2 16 2546 2530 79.0568 79.5 0.00499317 0.0125526 2022-03-10T01:31:48.125 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 3 16 3798 3782 78.7714 78.25 0.00967026 0.0126721 2022-03-10T01:31:48.481 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:31:49.125 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 4 16 4958 4942 77.2021 72.5 0.0161947 0.0129151 2022-03-10T01:31:49.316 INFO:teuthology.orchestra.run.smithi186.stderr:+ orig_num_osd=4 2022-03-10T01:31:49.316 INFO:teuthology.orchestra.run.smithi186.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-10T01:31:49.317 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-10T01:31:49.443 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:31:50.125 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 5 16 5915 5899 73.7234 59.8125 0.00524777 0.0133144 2022-03-10T01:31:51.125 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 6 16 6954 6938 72.2579 64.9375 0.0112116 0.0138093 2022-03-10T01:31:52.125 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 7 16 8004 7988 71.3096 65.625 0.00437733 0.0138303 2022-03-10T01:31:52.276 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:31:52.276178 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T01:31:52.276 INFO:tasks.rook.operator.smithi186.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-10T01:31:52.277 INFO:tasks.rook.operator.smithi186.stdout: Storage: v1.StorageScopeSpec{ 2022-03-10T01:31:52.277 INFO:tasks.rook.operator.smithi186.stdout: ... // 3 identical fields 2022-03-10T01:31:52.277 INFO:tasks.rook.operator.smithi186.stdout: Config: nil, 2022-03-10T01:31:52.277 INFO:tasks.rook.operator.smithi186.stdout: Selection: {}, 2022-03-10T01:31:52.278 INFO:tasks.rook.operator.smithi186.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T01:31:52.278 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:31:52.278 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:31:52.279 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:31:52.279 INFO:tasks.rook.operator.smithi186.stdout:- { 2022-03-10T01:31:52.279 INFO:tasks.rook.operator.smithi186.stdout:- Name: "smithi186-lv-4", 2022-03-10T01:31:52.279 INFO:tasks.rook.operator.smithi186.stdout:- Count: 1, 2022-03-10T01:31:52.280 INFO:tasks.rook.operator.smithi186.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-10T01:31:52.280 INFO:tasks.rook.operator.smithi186.stdout:- }, 2022-03-10T01:31:52.280 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-10T01:31:52.280 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-10T01:31:52.280 INFO:tasks.rook.operator.smithi186.stdout: Annotations: nil, 2022-03-10T01:31:52.281 INFO:tasks.rook.operator.smithi186.stdout: Labels: nil, 2022-03-10T01:31:52.281 INFO:tasks.rook.operator.smithi186.stdout: ... // 20 identical fields 2022-03-10T01:31:52.281 INFO:tasks.rook.operator.smithi186.stdout: } 2022-03-10T01:31:52.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:31:52.276259 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:31:52.905 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:31:52.911801 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:31:53.126 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 8 16 9096 9080 70.9267 68.25 0.0175073 0.0140849 2022-03-10T01:31:54.125 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 9 16 10004 9988 69.3509 56.75 0.0158824 0.0144044 2022-03-10T01:31:54.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:31:54.197962 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:31:55.125 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 10 16 10870 10854 67.8278 54.125 0.00699893 0.0146955 2022-03-10T01:31:56.125 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 11 16 10911 10895 61.8947 2.5625 0.00187953 0.0147244 2022-03-10T01:31:57.125 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 12 16 10911 10895 56.737 0 - 0.0147244 2022-03-10T01:31:58.126 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 13 16 10911 10895 52.3727 0 - 0.0147244 2022-03-10T01:31:59.126 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 14 16 10911 10895 48.6317 0 - 0.0147244 2022-03-10T01:32:00.126 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 15 16 10911 10895 45.3897 0 - 0.0147244 2022-03-10T01:32:01.126 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 16 16 10911 10895 42.5529 0 - 0.0147244 2022-03-10T01:32:01.507 INFO:teuthology.orchestra.run.smithi186.stdout:Success 2022-03-10T01:32:01.536 INFO:teuthology.orchestra.run.smithi186.stderr:+ removed_pv= 2022-03-10T01:32:01.536 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' '' = '' ']' 2022-03-10T01:32:01.537 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get pv 2022-03-10T01:32:01.537 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f1 -d ' ' 2022-03-10T01:32:01.538 INFO:teuthology.orchestra.run.smithi186.stderr:++ grep Released 2022-03-10T01:32:01.618 INFO:teuthology.orchestra.run.smithi186.stderr:+ removed_pv= 2022-03-10T01:32:01.619 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 3s 2022-03-10T01:32:02.126 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 17 16 10969 10953 40.2631 0.604167 0.00215512 0.0241266 2022-03-10T01:32:03.126 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 18 16 11006 10990 38.1548 2.3125 0.00272705 0.0240798 2022-03-10T01:32:04.127 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 19 16 11203 11187 36.7947 12.3125 0.0168801 0.027164 2022-03-10T01:32:04.619 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' '' = '' ']' 2022-03-10T01:32:04.620 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get pv 2022-03-10T01:32:04.621 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f1 -d ' ' 2022-03-10T01:32:04.621 INFO:teuthology.orchestra.run.smithi186.stderr:++ grep Released 2022-03-10T01:32:04.717 INFO:teuthology.orchestra.run.smithi186.stderr:+ removed_pv= 2022-03-10T01:32:04.718 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 3s 2022-03-10T01:32:05.127 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:32:05.134925+0000 min lat: 0.00129056 max lat: 8.77247 avg lat: 0.0266685 2022-03-10T01:32:05.127 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:32:05.127 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 20 16 12010 11994 37.4766 50.4375 0.0200703 0.0266685 2022-03-10T01:32:06.128 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 21 16 12633 12617 37.5438 38.9375 0.0106162 0.0266115 2022-03-10T01:32:07.128 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 22 16 13308 13292 37.7547 42.1875 0.0243766 0.0264667 2022-03-10T01:32:07.718 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' '' = '' ']' 2022-03-10T01:32:07.719 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get pv 2022-03-10T01:32:07.719 INFO:teuthology.orchestra.run.smithi186.stderr:++ grep Released 2022-03-10T01:32:07.720 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f1 -d ' ' 2022-03-10T01:32:07.840 INFO:teuthology.orchestra.run.smithi186.stderr:+ removed_pv= 2022-03-10T01:32:07.841 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 3s 2022-03-10T01:32:08.128 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 23 16 13995 13979 37.9798 42.9375 0.00216226 0.0259244 2022-03-10T01:32:08.655 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:08.657468 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-10T01:32:09.128 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 24 16 13995 13979 36.3975 0 - 0.0259244 2022-03-10T01:32:10.128 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 25 16 13995 13979 34.9416 0 - 0.0259244 2022-03-10T01:32:10.841 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' '' = '' ']' 2022-03-10T01:32:10.842 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get pv 2022-03-10T01:32:10.842 INFO:teuthology.orchestra.run.smithi186.stderr:++ grep Released 2022-03-10T01:32:10.843 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f1 -d ' ' 2022-03-10T01:32:11.128 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 26 16 13995 13979 33.5978 0 - 0.0259244 2022-03-10T01:32:11.217 INFO:teuthology.orchestra.run.smithi186.stderr:+ removed_pv=smithi186-lv-4 2022-03-10T01:32:11.217 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 3s 2022-03-10T01:32:12.128 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 27 16 13995 13979 32.3536 0 - 0.0259244 2022-03-10T01:32:13.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 28 16 13995 13979 31.1981 0 - 0.0259244 2022-03-10T01:32:14.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 29 16 13995 13979 30.1224 0 - 0.0259244 2022-03-10T01:32:14.218 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' smithi186-lv-4 = '' ']' 2022-03-10T01:32:14.218 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get pv smithi186-lv-4 -o 'jsonpath={.spec.local.path}' 2022-03-10T01:32:14.706 INFO:teuthology.orchestra.run.smithi186.stderr:+ target_path=/dev/vg_nvme/lv_4 2022-03-10T01:32:14.706 INFO:teuthology.orchestra.run.smithi186.stderr:++ echo smithi186-lv-4 2022-03-10T01:32:14.707 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f1 -d - 2022-03-10T01:32:14.707 INFO:teuthology.orchestra.run.smithi186.stderr:+ host=smithi186 2022-03-10T01:32:14.708 INFO:teuthology.orchestra.run.smithi186.stderr:+ toolbox ceph orch device zap smithi186 /dev/vg_nvme/lv_4 --force 2022-03-10T01:32:14.708 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi186 /dev/vg_nvme/lv_4 --force 2022-03-10T01:32:14.774 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:32:15.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 30 16 13995 13979 29.1184 0 - 0.0259244 2022-03-10T01:32:15.374 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_4 on smithi186 zapped 2022-03-10T01:32:15.394 INFO:teuthology.orchestra.run.smithi186.stderr:+ zap_completion=0 2022-03-10T01:32:15.394 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 0 = 0 ']' 2022-03-10T01:32:15.395 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-10T01:32:15.811 INFO:teuthology.orchestra.run.smithi186.stderr:+ zap_completion= 2022-03-10T01:32:15.811 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 3s 2022-03-10T01:32:16.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 31 16 13995 13979 28.1791 0 - 0.0259244 2022-03-10T01:32:17.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 32 16 13995 13979 27.2986 0 - 0.0259244 2022-03-10T01:32:18.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 33 16 14048 14032 26.5718 0.33125 0.00360223 0.0375674 2022-03-10T01:32:18.812 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' '' = 0 ']' 2022-03-10T01:32:18.812 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl patch pv smithi186-lv-4 -p '{"spec":{"claimRef": null}}' 2022-03-10T01:32:18.927 INFO:teuthology.orchestra.run.smithi186.stdout:persistentvolume/smithi186-lv-4 patched 2022-03-10T01:32:18.928 INFO:teuthology.orchestra.run.smithi186.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-10T01:32:18.929 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-10T01:32:19.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 34 16 14050 14034 25.794 0.125 0.00317122 0.0375625 2022-03-10T01:32:19.184 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:32:20.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 35 16 14050 14034 25.057 0 - 0.0375625 2022-03-10T01:32:21.110 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:21.114387 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:32:21.110 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:21.114412 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:32:21.110 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:21.114420 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:32:21.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 36 16 14050 14034 24.361 0 - 0.0375625 2022-03-10T01:32:22.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:22.020031 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:32:22.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 37 16 14050 14034 23.7027 0 - 0.0375625 2022-03-10T01:32:22.164 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:22.171630 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:32:22.164 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:22.171946 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:32:22.420 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:22.427914 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T01:32:22.421 INFO:tasks.rook.operator.smithi186.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-10T01:32:22.421 INFO:tasks.rook.operator.smithi186.stdout: Storage: v1.StorageScopeSpec{ 2022-03-10T01:32:22.421 INFO:tasks.rook.operator.smithi186.stdout: ... // 3 identical fields 2022-03-10T01:32:22.422 INFO:tasks.rook.operator.smithi186.stdout: Config: nil, 2022-03-10T01:32:22.422 INFO:tasks.rook.operator.smithi186.stdout: Selection: {}, 2022-03-10T01:32:22.422 INFO:tasks.rook.operator.smithi186.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T01:32:22.422 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:32:22.423 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:32:22.423 INFO:tasks.rook.operator.smithi186.stdout: {Name: "smithi186-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi186-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:32:22.423 INFO:tasks.rook.operator.smithi186.stdout:+ { 2022-03-10T01:32:22.424 INFO:tasks.rook.operator.smithi186.stdout:+ Name: "smithi186-lv-4", 2022-03-10T01:32:22.424 INFO:tasks.rook.operator.smithi186.stdout:+ Count: 1, 2022-03-10T01:32:22.424 INFO:tasks.rook.operator.smithi186.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-10T01:32:22.425 INFO:tasks.rook.operator.smithi186.stdout:+ }, 2022-03-10T01:32:22.425 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-10T01:32:22.425 INFO:tasks.rook.operator.smithi186.stdout: }, 2022-03-10T01:32:22.425 INFO:tasks.rook.operator.smithi186.stdout: Annotations: nil, 2022-03-10T01:32:22.425 INFO:tasks.rook.operator.smithi186.stdout: Labels: nil, 2022-03-10T01:32:22.426 INFO:tasks.rook.operator.smithi186.stdout: ... // 20 identical fields 2022-03-10T01:32:22.426 INFO:tasks.rook.operator.smithi186.stdout: } 2022-03-10T01:32:22.427 INFO:teuthology.orchestra.run.smithi186.stdout:['Success'] 2022-03-10T01:32:22.446 INFO:teuthology.orchestra.run.smithi186.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-10T01:32:22.512 INFO:teuthology.orchestra.run.smithi186.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-10T01:32:22.566 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-10T01:32:22.567 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=0 2022-03-10T01:32:22.567 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 0 '!=' 4 ']' 2022-03-10T01:32:22.567 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T01:32:22.572 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-10T01:32:22.573 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-10T01:32:22.573 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:32:22.650 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:32:22.841 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:22.846786 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:32:23.026 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:23.034028 I | op-mon: start running mons 2022-03-10T01:32:23.080 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:23.088168 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:32:23.094 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:23.101702 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:32:23.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:23.108711 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:32:23.103 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:23.108888 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:32:23.129 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 38 16 14050 14034 23.0789 0 - 0.0375625 2022-03-10T01:32:23.288 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-10T01:32:23.289 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-10T01:32:24.130 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 39 16 14050 14034 22.4872 0 - 0.0375625 2022-03-10T01:32:24.252 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:24.260423 I | op-mon: targeting the mon count 1 2022-03-10T01:32:24.967 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:24.972007 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-10T01:32:24.967 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:24.972091 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:32:25.044 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:25.049482 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:32:25.130 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:32:25.138128+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0375625 2022-03-10T01:32:25.130 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:32:25.130 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 40 16 14050 14034 21.925 0 - 0.0375625 2022-03-10T01:32:25.746 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:25.754155 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:32:26.130 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 41 16 14121 14105 21.4985 0.633929 0.00221584 0.0464386 2022-03-10T01:32:26.283 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:26.290178 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-10T01:32:26.288 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:26.296412 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T01:32:27.130 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 42 16 14121 14105 20.9864 0 - 0.0464386 2022-03-10T01:32:28.130 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 43 16 14121 14105 20.4984 0 - 0.0464386 2022-03-10T01:32:28.293 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:28.300788 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T01:32:29.130 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 44 16 14121 14105 20.0325 0 - 0.0464386 2022-03-10T01:32:30.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 45 16 14121 14105 19.5874 0 - 0.0464386 2022-03-10T01:32:30.298 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:32:30.305825 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-10T01:32:31.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 46 16 14121 14105 19.1616 0 - 0.0464386 2022-03-10T01:32:32.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 47 16 14121 14105 18.7539 0 - 0.0464386 2022-03-10T01:32:33.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 48 16 14121 14105 18.3632 0 - 0.0464386 2022-03-10T01:32:34.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 49 16 14138 14122 18.0101 0.132812 0.0100683 0.0514647 2022-03-10T01:32:35.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 50 16 14232 14216 17.7674 5.875 0.00173802 0.0552733 2022-03-10T01:32:36.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 51 16 14232 14216 17.4191 0 - 0.0552733 2022-03-10T01:32:37.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 52 16 14232 14216 17.0841 0 - 0.0552733 2022-03-10T01:32:38.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 53 16 14232 14216 16.7617 0 - 0.0552733 2022-03-10T01:32:39.131 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 54 16 14232 14216 16.4514 0 - 0.0552733 2022-03-10T01:32:40.132 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 55 16 14232 14216 16.1522 0 - 0.0552733 2022-03-10T01:32:41.132 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 56 16 14232 14216 15.8638 0 - 0.0552733 2022-03-10T01:32:42.136 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 57 16 14291 14275 15.649 0.526786 0.00267456 0.0616299 2022-03-10T01:32:43.136 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 58 16 14291 14275 15.3792 0 - 0.0616299 2022-03-10T01:32:44.136 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 59 16 14339 14323 15.1694 1.5 0.00260846 0.0647822 2022-03-10T01:32:45.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:32:45.145199+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0647822 2022-03-10T01:32:45.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:32:45.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 60 16 14339 14323 14.9166 0 - 0.0647822 2022-03-10T01:32:46.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 61 16 14339 14323 14.6721 0 - 0.0647822 2022-03-10T01:32:47.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 62 16 14339 14323 14.4355 0 - 0.0647822 2022-03-10T01:32:48.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 63 16 14339 14323 14.2064 0 - 0.0647822 2022-03-10T01:32:49.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 64 16 14339 14323 13.9844 0 - 0.0647822 2022-03-10T01:32:50.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 65 16 14381 14365 13.8097 0.4375 0.00225498 0.0674172 2022-03-10T01:32:51.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 66 16 14381 14365 13.6005 0 - 0.0674172 2022-03-10T01:32:52.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 67 16 14489 14473 13.4982 3.375 0.0118063 0.0740671 2022-03-10T01:32:53.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 68 16 15282 15266 14.0285 49.5625 0.00895148 0.0712706 2022-03-10T01:32:53.289 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-10T01:32:53.289 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T01:32:53.290 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-10T01:32:53.291 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-10T01:32:53.291 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:32:53.291 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-10T01:32:53.846 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:32:54.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 69 16 15948 15932 14.4283 41.625 0.00582301 0.0692299 2022-03-10T01:32:54.990 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-10T01:32:54.991 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-10T01:32:55.137 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 70 16 16854 16838 15.031 56.625 0.0157903 0.0665168 2022-03-10T01:32:56.138 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 71 16 17866 17850 15.71 63.25 0.00915083 0.0636405 2022-03-10T01:32:57.139 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 72 16 18864 18848 16.3577 62.375 0.012047 0.0611213 2022-03-10T01:32:58.139 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 73 16 19466 19450 16.649 37.625 0.0493186 0.0596456 2022-03-10T01:32:59.139 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 74 16 19466 19450 16.424 0 - 0.0596456 2022-03-10T01:33:00.139 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 75 16 20042 20026 16.6849 18 0.0102557 0.0599212 2022-03-10T01:33:01.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 76 16 20964 20948 17.2235 57.625 0.0620501 0.0580483 2022-03-10T01:33:02.139 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 77 16 21561 21545 17.4843 37.3125 0.0105463 0.0570538 2022-03-10T01:33:03.139 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 78 16 21561 21545 17.2602 0 - 0.0570538 2022-03-10T01:33:04.139 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 79 16 21561 21545 17.0417 0 - 0.0570538 2022-03-10T01:33:05.139 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:33:05.148266+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0577093 2022-03-10T01:33:05.140 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:33:05.140 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 80 16 22197 22181 17.3255 13.25 0.0140363 0.0577093 2022-03-10T01:33:06.140 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 81 15 23211 23196 17.8946 63.4375 0.0300671 0.0558645 2022-03-10T01:33:07.140 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 82 16 24067 24051 18.328 53.4375 0.0205355 0.0545313 2022-03-10T01:33:08.140 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 83 16 24599 24583 18.5077 33.25 0.026372 0.0538331 2022-03-10T01:33:09.140 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 84 16 25140 25124 18.6898 33.8125 0.00961369 0.0534955 2022-03-10T01:33:10.140 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 85 16 26046 26030 19.136 56.625 0.0146065 0.0522502 2022-03-10T01:33:11.141 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 86 16 26922 26906 19.5498 54.75 0.00860863 0.0511406 2022-03-10T01:33:12.141 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 87 16 27562 27546 19.7848 40 0.0151292 0.0505308 2022-03-10T01:33:13.141 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 88 16 28255 28239 20.0521 43.3125 0.0154235 0.0497499 2022-03-10T01:33:14.141 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 89 16 28679 28663 20.1245 26.5 0.0207084 0.0496514 2022-03-10T01:33:15.141 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 90 16 29351 29335 20.3675 42 0.00737214 0.049087 2022-03-10T01:33:16.141 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 91 16 30262 30246 20.7693 56.9375 0.0124708 0.0481361 2022-03-10T01:33:17.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 92 16 31095 31079 21.1093 52.0625 0.024361 0.0473636 2022-03-10T01:33:18.141 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 93 16 31628 31612 21.2405 33.3125 0.119794 0.0470665 2022-03-10T01:33:19.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 94 16 32525 32509 21.6109 56.0625 0.0134672 0.0462636 2022-03-10T01:33:20.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 95 16 33175 33159 21.811 40.625 0.0443611 0.0457991 2022-03-10T01:33:21.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 96 16 33918 33902 22.0674 46.4375 0.0205943 0.0453066 2022-03-10T01:33:22.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 97 16 34685 34669 22.334 47.9375 0.0246439 0.0447665 2022-03-10T01:33:23.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 98 16 35075 35059 22.3548 24.375 0.117047 0.0447263 2022-03-10T01:33:24.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 99 16 35835 35819 22.6087 47.5 0.0119744 0.0442128 2022-03-10T01:33:24.991 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-10T01:33:24.992 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-10T01:33:24.993 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T01:33:24.993 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-10T01:33:24.993 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:33:24.993 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-10T01:33:25.094 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:33:25.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:33:25.151188+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0436795 2022-03-10T01:33:25.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:33:25.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 100 16 36644 36628 22.8882 50.5625 0.0227739 0.0436795 2022-03-10T01:33:25.789 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-10T01:33:25.789 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-10T01:33:26.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 101 16 37426 37410 23.1454 48.875 0.0159711 0.0431964 2022-03-10T01:33:27.142 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 102 16 37992 37976 23.2653 35.375 0.020023 0.0429738 2022-03-10T01:33:28.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 103 16 38870 38854 23.5721 54.875 0.0145642 0.0424151 2022-03-10T01:33:29.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 104 16 39643 39627 23.8099 48.3125 0.0142804 0.0419914 2022-03-10T01:33:30.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 105 16 40276 40260 23.9599 39.5625 0.0149222 0.0417281 2022-03-10T01:33:31.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 106 16 40962 40946 24.1383 42.875 0.0122632 0.0414195 2022-03-10T01:33:32.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 107 16 41741 41725 24.3677 48.6875 0.0122986 0.0410283 2022-03-10T01:33:33.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 108 16 42328 42312 24.4817 36.6875 0.0156606 0.0408363 2022-03-10T01:33:34.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 109 16 42915 42899 24.5936 36.6875 0.0261739 0.0406534 2022-03-10T01:33:35.027 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:35.005918 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:33:35.027 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:35.006004 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:33:35.027 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:35.006023 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:33:35.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 110 16 43596 43580 24.7569 42.5625 0.0141645 0.0403843 2022-03-10T01:33:35.734 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:35.742939 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:33:36.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 111 16 44211 44195 24.8801 38.4375 0.0160002 0.0401834 2022-03-10T01:33:36.746 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:36.753693 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:33:36.749 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:36.754259 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:33:37.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 112 16 44829 44813 25.0028 38.625 0.0187242 0.0399881 2022-03-10T01:33:37.521 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:37.527273 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:33:37.656 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:37.661860 I | op-mon: start running mons 2022-03-10T01:33:37.688 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:37.694724 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:33:37.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:37.739471 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:33:37.784 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:37.792604 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:33:37.785 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:37.792803 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:33:38.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 113 16 45445 45429 25.1222 38.5 0.0285395 0.0397983 2022-03-10T01:33:38.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:38.835257 I | op-mon: targeting the mon count 1 2022-03-10T01:33:38.832 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:38.839181 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:33:39.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 114 16 45834 45818 25.1151 24.3125 0.0147868 0.0398091 2022-03-10T01:33:39.414 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:39.422401 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:33:39.417 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:39.422421 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:33:40.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 115 16 46285 46269 25.1418 28.1875 0.0199587 0.0397664 2022-03-10T01:33:40.348 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:40.356888 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:33:40.349 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:40.356938 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:33:40.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:40.834103 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:33:40.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:40.834126 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:33:41.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 116 16 46899 46883 25.2558 38.375 0.0291556 0.0395867 2022-03-10T01:33:41.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:41.427253 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:33:41.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:41.427279 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:33:42.055 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:42.063299 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:33:42.055 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:42.063322 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T01:33:42.143 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 117 16 47556 47540 25.3909 41.0625 0.0119761 0.0393741 2022-03-10T01:33:42.616 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:42.624916 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T01:33:42.617 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:42.624941 I | op-mon: checking for basic quorum with existing mons 2022-03-10T01:33:42.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:42.713508 I | op-mon: mon "a" endpoint is [v2:10.107.13.124:3300,v1:10.107.13.124:6789] 2022-03-10T01:33:42.724 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:42.732326 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:33:42.773 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:42.781370 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:33:42.777 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:42.781570 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:33:43.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:43.072235 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T01:33:43.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:43.084137 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T01:33:43.076 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:43.084161 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:33:43.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 118 16 48211 48195 25.5226 40.9375 0.00983958 0.0391725 2022-03-10T01:33:43.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:43.468112 I | op-mon: mons running: [a] 2022-03-10T01:33:44.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 119 16 48850 48834 25.6437 39.9375 0.0167266 0.0389852 2022-03-10T01:33:44.258 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:44.266090 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:33:44.258 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:44.266113 I | op-mon: mons created: 1 2022-03-10T01:33:44.975 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:44.982984 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:33:44.985 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:44.993787 I | op-mon: mons running: [a] 2022-03-10T01:33:45.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:33:45.153073+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0387866 2022-03-10T01:33:45.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:33:45.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 120 16 49513 49497 25.7753 41.4375 0.0111995 0.0387866 2022-03-10T01:33:45.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:45.701143 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:33:45.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:45.701177 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T01:33:46.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 121 16 50022 50006 25.8251 31.8125 0.0220118 0.0386468 2022-03-10T01:33:46.245 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:46.253804 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T01:33:47.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 122 16 50612 50596 25.9157 36.875 0.0362598 0.0385772 2022-03-10T01:33:47.169 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:47.177445 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T01:33:47.872 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:47.876014 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T01:33:48.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 123 16 51249 51233 26.0286 39.8125 0.0226394 0.0384092 2022-03-10T01:33:48.802 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:48.770706 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T01:33:48.802 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:48.770725 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T01:33:49.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 124 16 51680 51664 26.0359 26.9375 0.0305094 0.0383134 2022-03-10T01:33:49.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:49.494213 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T01:33:50.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 125 16 52148 52132 26.0616 29.25 0.030651 0.0383608 2022-03-10T01:33:50.255 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:50.261787 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T01:33:50.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:50.261816 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T01:33:50.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:50.261821 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T01:33:51.077 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:51.085074 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T01:33:51.134 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:51.141585 I | op-mgr: start running mgr 2022-03-10T01:33:51.135 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:51.141818 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T01:33:51.144 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 126 16 52784 52768 26.1702 39.75 0.0350781 0.0382034 2022-03-10T01:33:51.993 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.000528 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T01:33:52.002 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.010537 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T01:33:52.006 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.013273 I | op-mgr: setting services to point to mgr "a" 2022-03-10T01:33:52.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 127 16 53346 53330 26.2407 35.125 0.0183115 0.0380973 2022-03-10T01:33:52.166 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.174466 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T01:33:52.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.174491 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T01:33:52.209 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.203863 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T01:33:52.209 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.206854 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-1" 2022-03-10T01:33:52.209 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.206884 I | op-osd: OSD PVC "smithi186-lv-1-data-0c49qw" already exists 2022-03-10T01:33:52.209 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.206895 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-2" 2022-03-10T01:33:52.210 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.206905 I | op-osd: OSD PVC "smithi186-lv-2-data-02p4st" already exists 2022-03-10T01:33:52.210 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.206911 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-3" 2022-03-10T01:33:52.210 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.206919 I | op-osd: OSD PVC "smithi186-lv-3-data-0b5pdc" already exists 2022-03-10T01:33:52.210 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.206929 I | op-osd: creating 1 new PVCs for device set "smithi186-lv-4" 2022-03-10T01:33:52.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.217997 I | op-osd: successfully provisioned PVC "smithi186-lv-4-data-0g7q4z" 2022-03-10T01:33:52.260 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.267331 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0c49qw" 2022-03-10T01:33:52.265 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.272445 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-02p4st" 2022-03-10T01:33:52.265 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.272922 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:33:52.265 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.273037 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0g7q4z" 2022-03-10T01:33:52.330 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.336182 I | op-osd: started OSD provisioning job for PVC "smithi186-lv-4-data-0g7q4z" 2022-03-10T01:33:52.330 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.336207 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T01:33:52.330 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.336214 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T01:33:52.352 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:52.357191 I | op-osd: OSD orchestration status for PVC smithi186-lv-4-data-0g7q4z is "starting" 2022-03-10T01:33:53.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 128 16 53660 53644 26.189 19.625 0.0249943 0.0381756 2022-03-10T01:33:54.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 129 16 54140 54124 26.2185 30 0.0270918 0.0381022 2022-03-10T01:33:55.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 130 16 54670 54654 26.2716 33.125 0.00789736 0.038051 2022-03-10T01:33:55.790 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-10T01:33:55.791 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-10T01:33:55.791 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T01:33:55.792 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-10T01:33:55.792 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:33:55.793 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-10T01:33:55.882 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:55.843910 I | op-mgr: successful modules: dashboard 2022-03-10T01:33:55.882 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:55.846226 I | op-mgr: successful modules: prometheus 2022-03-10T01:33:55.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:55.847096 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T01:33:56.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 131 16 54985 54969 26.2213 19.6875 0.0183477 0.0380254 2022-03-10T01:33:56.537 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:56.545073 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-10T01:33:56.756 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:33:56.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:56.770828 I | op-osd: updating OSD 1 on PVC "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:33:56.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:56.770873 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:33:57.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 132 16 54985 54969 26.0227 0 - 0.0380254 2022-03-10T01:33:57.462 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:33:57.457005 I | op-mgr: successful modules: balancer 2022-03-10T01:33:57.684 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-10T01:33:57.684 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-10T01:33:58.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 133 16 55208 55192 25.9318 6.96875 0.0189557 0.0385525 2022-03-10T01:33:59.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 134 16 55752 55736 25.992 34 0.031168 0.0384638 2022-03-10T01:34:00.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 135 16 56241 56225 26.0258 30.5625 0.0232529 0.0384152 2022-03-10T01:34:00.339 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:00.347735 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-10T01:34:00.414 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:00.420599 I | op-osd: updating OSD 2 on PVC "smithi186-lv-2-data-02p4st" 2022-03-10T01:34:00.415 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:00.420629 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-02p4st" 2022-03-10T01:34:00.983 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:00.990295 I | op-mgr: successful modules: orchestrator modules 2022-03-10T01:34:01.145 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 136 16 56761 56745 26.0734 32.5 0.0176631 0.0383441 2022-03-10T01:34:02.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 137 16 57336 57320 26.1454 35.9375 0.0200225 0.0382375 2022-03-10T01:34:03.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 138 16 57894 57878 26.2086 34.875 0.0169526 0.0381456 2022-03-10T01:34:03.967 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:03.975501 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-10T01:34:04.100 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:04.107674 I | op-osd: updating OSD 3 on PVC "smithi186-lv-1-data-0c49qw" 2022-03-10T01:34:04.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:04.107699 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0c49qw" 2022-03-10T01:34:04.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 139 16 58416 58400 26.2547 32.625 0.0236487 0.0380808 2022-03-10T01:34:05.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:34:05.155026+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0379511 2022-03-10T01:34:05.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:34:05.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 140 16 59034 59018 26.3431 38.625 0.0191942 0.0379511 2022-03-10T01:34:06.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 141 16 59578 59562 26.3973 34 0.0123575 0.0378672 2022-03-10T01:34:07.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 142 16 60157 60141 26.4663 36.1875 0.0171782 0.0377724 2022-03-10T01:34:08.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 143 16 60738 60722 26.5351 36.3125 0.0204227 0.0376776 2022-03-10T01:34:09.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 144 16 61169 61153 26.5379 26.9375 0.0207649 0.0376742 2022-03-10T01:34:10.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 145 16 61775 61759 26.616 37.875 0.0223724 0.0375633 2022-03-10T01:34:11.146 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 146 16 62295 62279 26.6563 32.5 0.120314 0.0374982 2022-03-10T01:34:12.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 147 16 62830 62814 26.7024 33.4375 0.0158095 0.0374416 2022-03-10T01:34:13.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 148 16 63292 63276 26.7171 28.875 0.0203972 0.0374206 2022-03-10T01:34:14.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 149 16 63919 63903 26.8007 39.1875 0.0259779 0.0373049 2022-03-10T01:34:15.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 150 16 64505 64489 26.8662 36.625 0.0330681 0.0372122 2022-03-10T01:34:16.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 151 16 65124 65108 26.9445 38.6875 0.0215099 0.0371058 2022-03-10T01:34:17.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 152 16 65676 65660 26.9941 34.5 0.0287508 0.0370367 2022-03-10T01:34:18.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 153 16 66313 66297 27.0779 39.8125 0.015339 0.036923 2022-03-10T01:34:19.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 154 16 66825 66809 27.1098 32 0.0230123 0.0368783 2022-03-10T01:34:20.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 155 16 67275 67259 27.1164 28.125 0.0264961 0.0368439 2022-03-10T01:34:21.147 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 156 16 67736 67720 27.1272 28.8125 0.019088 0.0368513 2022-03-10T01:34:22.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 157 16 68180 68164 27.1312 27.75 0.0106816 0.0368493 2022-03-10T01:34:22.769 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:22.777616 I | op-osd: OSD orchestration status for node smithi186-lv-4-data-0g7q4z is "orchestrating" 2022-03-10T01:34:23.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 158 16 68654 68638 27.1469 29.625 0.0266556 0.0368282 2022-03-10T01:34:23.328 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:23.336455 I | op-osd: OSD orchestration status for PVC smithi186-lv-4-data-0g7q4z is "orchestrating" 2022-03-10T01:34:24.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 159 16 69136 69120 27.1656 30.125 0.0157513 0.0368025 2022-03-10T01:34:25.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:34:25.156926+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0367383 2022-03-10T01:34:25.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:34:25.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 160 16 69696 69680 27.2146 35 0.0217983 0.0367383 2022-03-10T01:34:26.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 161 16 70193 70177 27.2384 31.0625 0.0214135 0.0367034 2022-03-10T01:34:27.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 162 16 70672 70656 27.2551 29.9375 0.0102647 0.0366804 2022-03-10T01:34:27.684 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-10T01:34:27.685 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-10T01:34:27.686 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T01:34:27.687 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-10T01:34:27.687 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-10T01:34:27.687 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:34:27.780 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:34:28.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 163 16 71214 71198 27.2957 33.875 0.0320898 0.0366312 2022-03-10T01:34:28.512 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-10T01:34:28.512 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-10T01:34:29.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 164 16 71726 71710 27.3244 32 0.0232769 0.0365877 2022-03-10T01:34:30.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 165 16 72234 72218 27.3512 31.75 0.0364411 0.0365524 2022-03-10T01:34:31.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 166 16 72791 72775 27.3961 34.8125 0.026269 0.0364928 2022-03-10T01:34:31.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:31.817308 I | op-osd: OSD orchestration status for PVC smithi186-lv-4-data-0g7q4z is "completed" 2022-03-10T01:34:31.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:31.817330 I | op-osd: creating OSD 0 on PVC "smithi186-lv-4-data-0g7q4z" 2022-03-10T01:34:31.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:31.817342 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0g7q4z" 2022-03-10T01:34:32.148 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 167 16 73269 73253 27.4109 29.875 0.0549291 0.0364721 2022-03-10T01:34:33.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 168 16 73697 73681 27.407 26.75 0.0202333 0.0364779 2022-03-10T01:34:34.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 169 16 74120 74104 27.4012 26.4375 0.0181832 0.0364864 2022-03-10T01:34:34.526 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:34.534849 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T01:34:34.526 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:34.534874 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:34:34.670 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:34.677809 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:34:35.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 170 16 74539 74523 27.394 26.1875 0.0340175 0.0364878 2022-03-10T01:34:35.153 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:35.114031 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:34:35.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:34:35.620694 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:34:36.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 171 16 75040 75024 27.4169 31.3125 0.0362611 0.0364662 2022-03-10T01:34:37.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 172 16 75270 75254 27.3411 14.375 0.0186764 0.0365673 2022-03-10T01:34:38.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 173 16 75783 75767 27.3684 32.0625 0.0154093 0.0365306 2022-03-10T01:34:39.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 174 16 76271 76255 27.3864 30.5 0.0468156 0.0365019 2022-03-10T01:34:40.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 175 16 76670 76654 27.3724 24.9375 0.014929 0.0365014 2022-03-10T01:34:41.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 176 16 77211 77195 27.4089 33.8125 0.0186263 0.0364779 2022-03-10T01:34:42.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 177 16 77714 77698 27.4317 31.4375 0.0184049 0.0364454 2022-03-10T01:34:43.149 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 178 16 78219 78203 27.4549 31.5625 0.0316669 0.0364165 2022-03-10T01:34:44.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 179 16 78637 78621 27.4474 26.125 0.025505 0.0364189 2022-03-10T01:34:45.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:34:45.158984+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0363998 2022-03-10T01:34:45.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:34:45.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 180 16 79127 79111 27.465 30.625 0.0531597 0.0363998 2022-03-10T01:34:46.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 181 16 79594 79578 27.4745 29.1875 0.0292415 0.0363801 2022-03-10T01:34:47.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 182 16 80035 80019 27.475 27.5625 0.035271 0.0363836 2022-03-10T01:34:48.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 183 16 80523 80507 27.4915 30.5 0.0268179 0.0363664 2022-03-10T01:34:49.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 184 16 80988 80972 27.5 29.0625 0.0229722 0.0363567 2022-03-10T01:34:50.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 185 16 81498 81482 27.5237 31.875 0.022098 0.0363234 2022-03-10T01:34:51.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 186 16 81919 81903 27.5171 26.3125 0.0201511 0.0363337 2022-03-10T01:34:52.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 187 16 82323 82307 27.505 25.25 0.165472 0.0363274 2022-03-10T01:34:53.150 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 188 16 82756 82740 27.5026 27.0625 0.0337301 0.0363523 2022-03-10T01:34:54.151 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 189 16 83047 83031 27.4534 18.1875 0.0413309 0.036416 2022-03-10T01:34:55.151 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 190 16 83486 83470 27.4533 27.4375 0.0341329 0.0364169 2022-03-10T01:34:56.151 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 191 16 83879 83863 27.4381 24.5625 0.0757195 0.0364373 2022-03-10T01:34:57.151 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 192 16 84320 84304 27.4387 27.5625 0.0331752 0.0364348 2022-03-10T01:34:58.151 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 193 16 84829 84813 27.4614 31.8125 0.0185066 0.0364062 2022-03-10T01:34:58.512 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-10T01:34:58.513 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T01:34:58.513 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-10T01:34:58.514 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-10T01:34:58.514 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:34:58.515 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-10T01:34:58.617 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:34:59.151 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 194 16 85217 85201 27.4448 24.25 0.0368377 0.0364268 2022-03-10T01:34:59.923 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=3 2022-03-10T01:34:59.923 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-10T01:35:00.151 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 195 16 85612 85596 27.4307 24.6875 0.0596828 0.0364471 2022-03-10T01:35:00.713 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:00.720180 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:35:00.714 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:00.720204 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:35:00.714 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:00.720214 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:35:01.151 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 196 16 86074 86058 27.438 28.875 0.0301966 0.0364387 2022-03-10T01:35:01.255 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:01.264376 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:35:01.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:01.971551 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:35:01.964 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:01.971761 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:35:02.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 197 16 86526 86510 27.4421 28.25 0.0273965 0.0364313 2022-03-10T01:35:02.691 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:02.699793 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:35:02.847 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:02.855900 I | op-mon: start running mons 2022-03-10T01:35:02.870 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:02.878623 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:35:02.888 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:02.897127 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:35:02.903 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:02.911419 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:35:02.903 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:02.911616 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:35:03.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 198 16 86952 86936 27.438 26.625 0.0389579 0.0364365 2022-03-10T01:35:03.966 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:03.974433 I | op-mon: targeting the mon count 1 2022-03-10T01:35:03.976 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:03.981404 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:35:04.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 199 16 87322 87306 27.4163 23.125 0.0644274 0.0364647 2022-03-10T01:35:05.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:35:05.161250+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0364451 2022-03-10T01:35:05.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:35:05.153 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 200 16 87808 87792 27.431 30.375 0.00757806 0.0364451 2022-03-10T01:35:05.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:05.326893 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:35:05.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:05.326920 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:35:05.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:06.002751 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:35:05.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:06.002778 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:35:06.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 201 16 88214 88198 27.4208 25.375 0.0473325 0.0364613 2022-03-10T01:35:07.111 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:07.115853 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:35:07.111 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:07.115881 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:35:07.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 202 16 88644 88628 27.4181 26.875 0.0518203 0.0364646 2022-03-10T01:35:08.009 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:08.017749 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:35:08.009 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:08.017775 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:35:08.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 203 16 89025 89009 27.4003 23.8125 0.0609555 0.0364855 2022-03-10T01:35:08.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:08.971684 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-10T01:35:08.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:08.971707 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T01:35:09.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 204 16 89416 89400 27.3858 24.4375 0.0733394 0.0365062 2022-03-10T01:35:09.955 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:09.963752 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T01:35:09.955 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:09.963780 I | op-mon: checking for basic quorum with existing mons 2022-03-10T01:35:10.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 205 16 89878 89862 27.393 28.875 0.0171504 0.0364977 2022-03-10T01:35:11.050 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:11.046003 I | op-mon: mon "a" endpoint is [v2:10.107.13.124:3300,v1:10.107.13.124:6789] 2022-03-10T01:35:11.152 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 206 16 90297 90281 27.3872 26.1875 0.0199759 0.0365046 2022-03-10T01:35:11.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:11.674107 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.13.124:6789"]}] data:a=10.107.13.124:6789 mapping:{"node":{"a":{"Name":"smithi186","Hostname":"smithi186","Address":"172.21.15.186"}}} maxMonId:0] 2022-03-10T01:35:11.779 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:11.787911 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:35:11.802 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:11.804095 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:35:11.812 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:11.820697 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T01:35:11.821 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:11.829902 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T01:35:11.821 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:11.829933 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:35:11.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:11.836595 I | op-mon: mons running: [a] 2022-03-10T01:35:12.153 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 207 16 90720 90704 27.3825 26.4375 0.0434507 0.0365113 2022-03-10T01:35:12.828 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:12.821846 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:35:12.829 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:12.821907 I | op-mon: mons created: 1 2022-03-10T01:35:13.153 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 208 16 91101 91085 27.3654 23.8125 0.0457843 0.0365314 2022-03-10T01:35:13.605 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:13.612139 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:35:13.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:13.626980 I | op-mon: mons running: [a] 2022-03-10T01:35:14.153 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 209 16 91463 91447 27.3426 22.625 0.0736867 0.0365619 2022-03-10T01:35:14.226 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:14.235152 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:35:14.227 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:14.235182 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T01:35:15.116 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:15.125125 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T01:35:15.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 210 16 91818 91802 27.318 22.1875 0.0322805 0.0365972 2022-03-10T01:35:15.981 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:15.990016 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T01:35:16.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 211 16 92228 92212 27.31 25.625 0.00850308 0.036609 2022-03-10T01:35:16.672 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:16.680290 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T01:35:17.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 212 16 92596 92580 27.2897 23 0.0600227 0.0366344 2022-03-10T01:35:17.535 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:17.543841 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T01:35:17.535 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:17.543865 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T01:35:18.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 213 16 93007 92991 27.2821 25.6875 0.0405904 0.0366455 2022-03-10T01:35:18.380 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:18.389176 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T01:35:19.098 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:19.107092 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T01:35:19.099 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:19.107158 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T01:35:19.099 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:19.107180 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T01:35:19.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 214 16 93430 93414 27.2782 26.4375 0.0339941 0.0366494 2022-03-10T01:35:19.887 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:19.894031 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T01:35:19.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:19.930847 I | op-mgr: start running mgr 2022-03-10T01:35:19.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:19.930876 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T01:35:20.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 215 16 93829 93813 27.2673 24.9375 0.0293177 0.0366642 2022-03-10T01:35:20.784 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.792420 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T01:35:20.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.802748 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T01:35:20.802 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.805268 I | op-mgr: setting services to point to mgr "a" 2022-03-10T01:35:20.949 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.957293 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T01:35:20.949 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.957368 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T01:35:20.960 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.968781 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T01:35:20.977 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.985746 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-1" 2022-03-10T01:35:20.978 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.985780 I | op-osd: OSD PVC "smithi186-lv-1-data-0c49qw" already exists 2022-03-10T01:35:20.978 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.985790 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-2" 2022-03-10T01:35:20.979 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.985799 I | op-osd: OSD PVC "smithi186-lv-2-data-02p4st" already exists 2022-03-10T01:35:20.979 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.985806 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-3" 2022-03-10T01:35:20.980 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.985814 I | op-osd: OSD PVC "smithi186-lv-3-data-0b5pdc" already exists 2022-03-10T01:35:20.980 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.985824 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi186-lv-4" 2022-03-10T01:35:20.981 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:20.985833 I | op-osd: OSD PVC "smithi186-lv-4-data-0g7q4z" already exists 2022-03-10T01:35:20.992 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:21.000214 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0c49qw" 2022-03-10T01:35:20.992 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:21.000237 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-02p4st" 2022-03-10T01:35:20.993 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:21.000243 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:35:20.993 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:21.000249 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0g7q4z" 2022-03-10T01:35:20.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:21.000253 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T01:35:20.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:21.000257 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T01:35:21.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 216 16 94205 94189 27.2498 23.5 0.0269665 0.0366863 2022-03-10T01:35:21.917 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:21.926124 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T01:35:22.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 217 16 94415 94399 27.1847 13.125 0.0100736 0.0367293 2022-03-10T01:35:22.567 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:22.575753 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-10T01:35:22.571 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:22.580465 I | op-osd: updating OSD 0 on PVC "smithi186-lv-4-data-0g7q4z" 2022-03-10T01:35:22.572 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:22.580490 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-4-data-0g7q4z" 2022-03-10T01:35:23.048 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:23.057235 I | op-mgr: successful modules: prometheus 2022-03-10T01:35:23.049 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:23.057389 I | op-mgr: successful modules: dashboard 2022-03-10T01:35:23.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 218 16 94415 94399 27.0601 0 - 0.0367293 2022-03-10T01:35:23.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:23.327859 I | op-mgr: successful modules: balancer 2022-03-10T01:35:23.457 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:23.465894 I | op-mgr: successful modules: orchestrator modules 2022-03-10T01:35:23.842 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:23.850543 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-10T01:35:23.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:23.858571 I | op-osd: updating OSD 1 on PVC "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:35:23.850 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:23.858597 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-3-data-0b5pdc" 2022-03-10T01:35:24.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 219 16 94415 94399 26.9365 0 - 0.0367293 2022-03-10T01:35:25.154 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:35:25.163921+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0367293 2022-03-10T01:35:25.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:35:25.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 220 16 94415 94399 26.8141 0 - 0.0367293 2022-03-10T01:35:25.783 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:25.792424 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-10T01:35:25.791 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:25.799485 I | op-osd: updating OSD 2 on PVC "smithi186-lv-2-data-02p4st" 2022-03-10T01:35:25.791 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:25.799505 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-2-data-02p4st" 2022-03-10T01:35:26.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 221 16 94415 94399 26.6927 0 - 0.0367293 2022-03-10T01:35:27.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 222 16 94575 94559 26.6175 2 0.0612209 0.0375563 2022-03-10T01:35:28.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 223 16 94821 94805 26.5671 15.375 0.0304419 0.0376279 2022-03-10T01:35:28.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:28.263665 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-10T01:35:28.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:28.269810 I | op-osd: updating OSD 3 on PVC "smithi186-lv-1-data-0c49qw" 2022-03-10T01:35:28.264 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:28.269842 I | op-osd: OSD will have its main bluestore block on "smithi186-lv-1-data-0c49qw" 2022-03-10T01:35:29.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 224 16 95052 95036 26.513 14.4375 0.0315524 0.037706 2022-03-10T01:35:29.198 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:29.206841 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T01:35:29.208 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:35:29.206866 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:35:29.924 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 3 '!=' 4 ']' 2022-03-10T01:35:29.924 INFO:teuthology.orchestra.run.smithi186.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T01:35:29.924 INFO:teuthology.orchestra.run.smithi186.stdout:waiting for osd to come back up 2022-03-10T01:35:29.925 INFO:teuthology.orchestra.run.smithi186.stderr:++ toolbox ceph osd stat 2022-03-10T01:35:29.926 INFO:teuthology.orchestra.run.smithi186.stderr:++ cut -f3 -d ' ' 2022-03-10T01:35:29.926 INFO:teuthology.orchestra.run.smithi186.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:35:30.027 INFO:teuthology.orchestra.run.smithi186.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:35:30.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 225 16 95304 95288 26.4651 15.75 0.0687698 0.0377727 2022-03-10T01:35:30.804 INFO:teuthology.orchestra.run.smithi186.stderr:+ num_osd=4 2022-03-10T01:35:30.804 INFO:teuthology.orchestra.run.smithi186.stderr:+ sleep 30s 2022-03-10T01:35:31.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 226 16 95550 95534 26.416 15.375 0.0260157 0.0378435 2022-03-10T01:35:32.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 227 16 95771 95755 26.3605 13.8125 0.0158669 0.0378941 2022-03-10T01:35:33.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 228 16 95943 95927 26.292 10.75 0.0662 0.0380242 2022-03-10T01:35:34.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 229 16 96212 96196 26.2506 16.8125 0.0378561 0.0380824 2022-03-10T01:35:35.155 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 230 16 96485 96469 26.2107 17.0625 0.0945722 0.0381409 2022-03-10T01:35:36.156 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 231 16 96756 96740 26.1705 16.9375 0.032734 0.0381991 2022-03-10T01:35:37.156 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 232 16 97002 96986 26.124 15.375 0.0327631 0.0382679 2022-03-10T01:35:38.156 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 233 16 97275 97259 26.0851 17.0625 0.0915596 0.0383244 2022-03-10T01:35:39.156 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 234 16 97400 97384 26.007 7.8125 0.563874 0.0383913 2022-03-10T01:35:40.156 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 235 16 97645 97629 25.9615 15.3125 0.0509397 0.0385074 2022-03-10T01:35:41.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 236 16 97842 97826 25.9036 12.3125 0.0673598 0.0385942 2022-03-10T01:35:42.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 237 16 98103 98087 25.8632 16.3125 0.129088 0.0386518 2022-03-10T01:35:43.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 238 16 98359 98343 25.8217 16 0.121801 0.0387156 2022-03-10T01:35:44.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 239 16 98618 98602 25.7814 16.1875 0.0296998 0.0387754 2022-03-10T01:35:45.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:35:45.166374+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0388204 2022-03-10T01:35:45.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:35:45.158 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 240 16 98823 98807 25.7274 12.8125 0.0127976 0.0388204 2022-03-10T01:35:46.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 241 16 99015 98999 25.6704 12 0.0432006 0.0389442 2022-03-10T01:35:47.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 242 16 99260 99244 25.6276 15.3125 0.0910263 0.0390101 2022-03-10T01:35:48.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 243 16 99521 99505 25.5893 16.3125 0.0859953 0.039068 2022-03-10T01:35:49.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 244 16 99736 99720 25.5394 13.4375 0.210416 0.039136 2022-03-10T01:35:50.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 245 16 99945 99929 25.4885 13.0625 0.0558656 0.0392214 2022-03-10T01:35:51.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 246 16 100162 100146 25.44 13.5625 0.0922442 0.0392961 2022-03-10T01:35:52.157 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 247 16 100397 100381 25.3965 14.6875 0.0447273 0.0393643 2022-03-10T01:35:53.158 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 248 16 100497 100481 25.3193 6.25 0.00945367 0.0394168 2022-03-10T01:35:54.158 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 249 16 100497 100481 25.2176 0 - 0.0394168 2022-03-10T01:35:55.158 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 250 16 100588 100572 25.1395 2.84375 0.0805971 0.0397662 2022-03-10T01:35:56.158 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 251 16 100802 100786 25.0926 13.375 0.0448279 0.039838 2022-03-10T01:35:57.158 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 252 16 101020 101004 25.0471 13.625 0.0593464 0.0399118 2022-03-10T01:35:58.158 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 253 16 101244 101228 25.0034 14 0.0765695 0.0399786 2022-03-10T01:35:59.158 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 254 16 101475 101459 24.9618 14.4375 0.0306293 0.0400459 2022-03-10T01:36:00.159 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 255 16 101729 101713 24.9261 15.875 0.0538436 0.0401084 2022-03-10T01:36:00.804 INFO:teuthology.orchestra.run.smithi186.stderr:+ '[' 4 '!=' 4 ']' 2022-03-10T01:36:00.806 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-10T01:36:00.821 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch status 2022-03-10T01:36:01.163 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 256 16 101963 101947 24.8855 14.625 0.0476838 0.0401707 2022-03-10T01:36:02.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 257 16 102209 102193 24.8485 15.375 0.0649189 0.0402339 2022-03-10T01:36:03.163 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 258 16 102307 102291 24.7759 6.125 0.0355309 0.040348 2022-03-10T01:36:04.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 259 16 102535 102519 24.7352 14.25 0.0496791 0.0404147 2022-03-10T01:36:05.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:36:05.173296+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0404772 2022-03-10T01:36:05.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:36:05.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 260 16 102757 102741 24.6935 13.875 0.0401331 0.0404772 2022-03-10T01:36:06.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 261 16 102929 102913 24.6401 10.75 0.049713 0.0405725 2022-03-10T01:36:06.872 INFO:teuthology.orchestra.run.smithi186.stdout:Backend: rook 2022-03-10T01:36:06.872 INFO:teuthology.orchestra.run.smithi186.stdout:Available: Yes 2022-03-10T01:36:06.911 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ps 2022-03-10T01:36:07.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 262 16 103144 103128 24.5973 13.4375 0.129517 0.040645 2022-03-10T01:36:07.824 INFO:teuthology.orchestra.run.smithi186.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-10T01:36:07.824 INFO:teuthology.orchestra.run.smithi186.stdout:crashcollector.crash smithi186 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-10T01:36:07.824 INFO:teuthology.orchestra.run.smithi186.stdout:mgr.a smithi186 running (9m) 0s ago 9m - - 4b871cde5e1a 2022-03-10T01:36:07.825 INFO:teuthology.orchestra.run.smithi186.stdout:mon.a smithi186 running (10m) 0s ago 10m - - 4b871cde5e1a 2022-03-10T01:36:07.825 INFO:teuthology.orchestra.run.smithi186.stdout:osd.0 smithi186 running (93s) 0s ago 94s - - 4b871cde5e1a 2022-03-10T01:36:07.825 INFO:teuthology.orchestra.run.smithi186.stdout:osd.1 smithi186 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-10T01:36:07.825 INFO:teuthology.orchestra.run.smithi186.stdout:osd.2 smithi186 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-10T01:36:07.825 INFO:teuthology.orchestra.run.smithi186.stdout:osd.3 smithi186 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-10T01:36:07.852 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls 2022-03-10T01:36:08.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 263 16 103364 103348 24.556 13.75 0.0511039 0.0407119 2022-03-10T01:36:08.940 INFO:teuthology.orchestra.run.smithi186.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-10T01:36:08.940 INFO:teuthology.orchestra.run.smithi186.stdout:crash 1/1 0s ago 7m * 2022-03-10T01:36:08.940 INFO:teuthology.orchestra.run.smithi186.stdout:mgr 1/1 0s ago 9m count:1 2022-03-10T01:36:08.941 INFO:teuthology.orchestra.run.smithi186.stdout:mon 1/1 0s ago 10m count:1 2022-03-10T01:36:08.941 INFO:teuthology.orchestra.run.smithi186.stdout:osd 8 0s ago 7m 2022-03-10T01:36:08.941 INFO:teuthology.orchestra.run.smithi186.stdout:osd.all-available-devices 0 0s ago - * 2022-03-10T01:36:08.980 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch host ls 2022-03-10T01:36:09.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 264 16 103560 103544 24.5094 12.25 0.0100708 0.0407696 2022-03-10T01:36:09.716 INFO:teuthology.orchestra.run.smithi186.stdout:HOST ADDR LABELS STATUS 2022-03-10T01:36:09.717 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 172.21.15.186/smithi186 2022-03-10T01:36:09.717 INFO:teuthology.orchestra.run.smithi186.stdout:1 hosts in cluster 2022-03-10T01:36:09.746 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch device ls 2022-03-10T01:36:10.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 265 16 103711 103695 24.4525 9.4375 0.0529001 0.0408842 2022-03-10T01:36:10.820 INFO:teuthology.orchestra.run.smithi186.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-10T01:36:10.820 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T01:36:10.821 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T01:36:10.821 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T01:36:10.821 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T01:36:10.889 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch apply rgw foo 2022-03-10T01:36:11.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 266 16 103953 103937 24.4175 15.125 0.0416957 0.040943 2022-03-10T01:36:11.869 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:11.876433 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-10T01:36:11.869 INFO:teuthology.orchestra.run.smithi186.stdout:Created 2022-03-10T01:36:11.902 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch apply mds foo 2022-03-10T01:36:11.980 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:11.978469 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:36:12.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 267 16 104171 104155 24.3771 13.625 0.0243054 0.0410048 2022-03-10T01:36:12.796 INFO:teuthology.orchestra.run.smithi186.stdout:Created 2022-03-10T01:36:12.797 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:12.800275 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-10T01:36:12.812 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:12.820755 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:36:12.818 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:12.824650 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:36:12.853 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch apply rbd-mirror 2022-03-10T01:36:13.164 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 268 16 104373 104357 24.3332 12.625 0.0621115 0.0410836 2022-03-10T01:36:13.878 INFO:teuthology.orchestra.run.smithi186.stdout:Success 2022-03-10T01:36:13.944 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch apply nfs foo --port 12777 2022-03-10T01:36:13.967 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:13.969932 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:36:14.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 269 16 104542 104526 24.282 10.5625 0.0959353 0.0411631 2022-03-10T01:36:14.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:14.630220 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:36:14.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:14.739265 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:36:14.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:14.739292 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:36:14.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:14.875977 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-10T01:36:14.870 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:14.876965 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-10T01:36:14.871 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:14.877232 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-10T01:36:15.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 270 16 104693 104677 24.227 9.4375 0.219066 0.0412588 2022-03-10T01:36:15.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:15.884883 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:36:16.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 271 16 104881 104865 24.181 11.75 0.234651 0.0413347 2022-03-10T01:36:17.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 272 16 105029 105013 24.1261 9.25 0.129427 0.0414365 2022-03-10T01:36:17.364 INFO:teuthology.orchestra.run.smithi186.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-10T01:36:17.364 INFO:teuthology.orchestra.run.smithi186.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-10T01:36:17.408 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2022-03-10T01:36:17.421 INFO:tasks.radosbench:joining radosbench (timing out after 11100s) 2022-03-10T01:36:17.422 INFO:teuthology.orchestra.run:waiting for 11100 2022-03-10T01:36:17.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:17.628904 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-10T01:36:18.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 273 16 105169 105153 24.0698 8.75 0.0837927 0.0415285 2022-03-10T01:36:19.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 274 16 105226 105210 23.9949 3.5625 0.775176 0.0416507 2022-03-10T01:36:20.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 275 16 105399 105383 23.947 10.8125 0.131443 0.0417406 2022-03-10T01:36:21.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 276 16 105593 105577 23.9042 12.125 0.0753955 0.0418173 2022-03-10T01:36:22.048 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:22.053108 I | ceph-file-controller: creating filesystem "foo" 2022-03-10T01:36:22.166 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 277 16 105821 105805 23.8693 14.25 0.0661722 0.0418785 2022-03-10T01:36:23.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 278 16 106007 105991 23.8253 11.625 0.0888291 0.0419605 2022-03-10T01:36:24.165 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 279 16 106157 106141 23.7735 9.375 0.242244 0.0420533 2022-03-10T01:36:25.166 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:36:25.175450+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.042109 2022-03-10T01:36:25.166 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:36:25.167 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 280 16 106385 106369 23.7394 14.25 0.0547115 0.042109 2022-03-10T01:36:26.166 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 281 16 106586 106570 23.6997 12.5625 0.0625456 0.0421853 2022-03-10T01:36:27.166 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 282 16 106804 106788 23.6639 13.625 0.0324386 0.0422407 2022-03-10T01:36:28.166 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 283 16 106986 106970 23.6205 11.375 0.0937526 0.0423138 2022-03-10T01:36:29.166 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 284 16 107164 107148 23.5765 11.125 0.203004 0.0424048 2022-03-10T01:36:30.166 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 285 16 107362 107346 23.5372 12.375 0.147026 0.0424723 2022-03-10T01:36:31.167 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 286 16 107568 107552 23.4999 12.875 0.0520964 0.0425381 2022-03-10T01:36:32.148 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:32.151344 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.index" 2022-03-10T01:36:32.154 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:32.161573 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.control" 2022-03-10T01:36:32.168 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 287 16 107700 107684 23.4466 8.25 0.131051 0.0426241 2022-03-10T01:36:32.212 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:32.215254 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.log" 2022-03-10T01:36:33.168 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 288 16 107842 107826 23.396 8.875 0.0458227 0.0427231 2022-03-10T01:36:34.168 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 289 16 108037 108021 23.3573 12.1875 0.0814116 0.0428022 2022-03-10T01:36:35.168 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 290 16 108273 108257 23.3276 14.75 0.0391744 0.0428568 2022-03-10T01:36:36.169 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 291 16 108535 108519 23.3037 16.375 0.106262 0.0429006 2022-03-10T01:36:36.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:36.505458 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.non-ec" 2022-03-10T01:36:36.539 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:36.547920 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.meta" 2022-03-10T01:36:36.588 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:36.566754 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-metadata" 2022-03-10T01:36:37.169 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 292 16 108699 108683 23.259 10.25 0.235268 0.0429793 2022-03-10T01:36:38.169 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 293 16 108888 108872 23.2199 11.8125 0.100313 0.0430558 2022-03-10T01:36:39.169 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 294 16 109086 109070 23.183 12.375 0.023298 0.0431201 2022-03-10T01:36:40.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:40.097624 I | cephclient: creating replicated pool foo.rgw.control succeeded 2022-03-10T01:36:40.100 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:40.104947 I | cephclient: creating replicated pool foo.rgw.log succeeded 2022-03-10T01:36:40.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:40.124677 I | cephclient: creating replicated pool foo.rgw.buckets.index succeeded 2022-03-10T01:36:40.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:40.129921 I | cephclient: creating replicated pool foo-metadata succeeded 2022-03-10T01:36:40.169 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 295 16 109318 109302 23.1536 14.5 0.19223 0.0431709 2022-03-10T01:36:41.169 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 296 16 109517 109501 23.1174 12.4375 0.0759441 0.0432472 2022-03-10T01:36:42.169 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 297 16 109739 109723 23.0863 13.875 0.0458924 0.0433045 2022-03-10T01:36:43.170 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 298 16 109928 109912 23.0484 11.8125 0.0600736 0.0433733 2022-03-10T01:36:43.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:43.187939 I | cephclient: creating replicated pool foo.rgw.meta succeeded 2022-03-10T01:36:43.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:43.188875 I | cephclient: creating replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-10T01:36:43.625 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:43.632707 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:36:43.625 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:43.632734 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:36:43.716 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:43.723184 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:36:44.170 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 299 16 110128 110112 23.0131 12.5 0.0932544 0.0434416 2022-03-10T01:36:45.170 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:36:45.179573+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0435029 2022-03-10T01:36:45.170 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:36:45.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 300 16 110347 110331 22.982 13.6875 0.0277783 0.0435029 2022-03-10T01:36:46.170 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 301 16 110542 110526 22.9461 12.1875 0.0426669 0.0435691 2022-03-10T01:36:47.170 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 302 16 110728 110712 22.9087 11.625 0.0246764 0.0436416 2022-03-10T01:36:47.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:47.945343 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:36:48.170 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 303 16 110951 110935 22.879 13.9375 0.0289832 0.0436978 2022-03-10T01:36:48.498 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:48.501226 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:36:48.498 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:48.501252 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:36:49.077 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:49.086079 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:36:49.170 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 304 16 111167 111151 22.8482 13.5 0.0625822 0.04375 2022-03-10T01:36:49.248 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:49.253869 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-data0" 2022-03-10T01:36:50.170 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 305 16 111281 111265 22.7966 7.125 0.0816325 0.0438522 2022-03-10T01:36:51.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 306 16 111466 111450 22.7599 11.5625 0.0853889 0.0439276 2022-03-10T01:36:52.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 307 16 111705 111689 22.7344 14.9375 0.0467889 0.0439731 2022-03-10T01:36:52.184 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:52.192961 I | cephclient: creating replicated pool foo-data0 succeeded 2022-03-10T01:36:52.184 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:52.192995 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-10T01:36:53.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 308 16 111898 111882 22.6998 12.0625 0.0405492 0.0440435 2022-03-10T01:36:53.920 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:53.928837 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:36:53.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:53.944384 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:36:54.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 309 16 112069 112053 22.6609 10.6875 0.252907 0.0441137 2022-03-10T01:36:54.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:54.469864 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-10T01:36:55.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 310 16 112220 112204 22.6183 9.4375 0.169851 0.0442009 2022-03-10T01:36:55.199 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:55.208403 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-10T01:36:56.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 311 16 112414 112398 22.5845 12.125 0.0284208 0.0442628 2022-03-10T01:36:57.057 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:57.057366 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:36:57.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 312 16 112592 112576 22.5478 11.125 0.0285249 0.0443367 2022-03-10T01:36:58.014 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:58.022908 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:36:58.015 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:58.022930 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:36:58.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 313 16 112823 112807 22.5219 14.4375 0.0498226 0.0443867 2022-03-10T01:36:59.052 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:59.061385 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-10T01:36:59.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 314 16 113000 112984 22.4854 11.0625 0.0825142 0.0444584 2022-03-10T01:36:59.196 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:36:59.205030 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:37:00.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 315 16 113137 113121 22.4412 8.5625 0.173896 0.0445401 2022-03-10T01:37:00.468 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:00.472537 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-10T01:37:01.171 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 316 16 113283 113267 22.399 9.125 0.0496528 0.0446322 2022-03-10T01:37:01.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:01.706732 I | op-mds: setting mds config flags 2022-03-10T01:37:01.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:01.706760 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T01:37:02.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 317 16 113481 113465 22.3674 12.375 0.1267 0.0446924 2022-03-10T01:37:02.743 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:02.751304 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T01:37:02.797 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:02.806182 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-10T01:37:03.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 318 15 113667 113652 22.3338 11.6875 0.024067 0.0447652 2022-03-10T01:37:03.491 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:03.500363 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:37:03.524 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:03.529997 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:37:03.709 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:03.710047 I | op-mds: setting mds config flags 2022-03-10T01:37:03.710 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:03.710137 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T01:37:04.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 319 16 113782 113766 22.2862 7.125 0.348707 0.0448396 2022-03-10T01:37:05.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:37:05.181549+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0449602 2022-03-10T01:37:05.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:37:05.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 320 16 113885 113869 22.2366 6.4375 0.0907929 0.0449602 2022-03-10T01:37:05.722 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:05.724992 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T01:37:06.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 321 16 114093 114077 22.2078 13 0.0431141 0.0450166 2022-03-10T01:37:06.726 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:06.734255 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:37:07.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 322 16 114273 114257 22.1738 11.25 0.069878 0.0450817 2022-03-10T01:37:08.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 323 16 114512 114496 22.1514 14.9375 0.272584 0.0451204 2022-03-10T01:37:08.297 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:08.305336 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:37:08.297 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:08.305355 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:37:09.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 324 16 114677 114661 22.1149 10.3125 0.0493515 0.0452094 2022-03-10T01:37:09.365 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:09.304199 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:37:10.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 325 16 114904 114888 22.0905 14.1875 0.0625371 0.0452555 2022-03-10T01:37:11.172 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 326 16 114958 114942 22.0331 3.375 0.341887 0.0453723 2022-03-10T01:37:12.173 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 327 16 115220 115204 22.0158 16.375 0.190957 0.0453913 2022-03-10T01:37:13.173 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 328 16 115391 115375 21.9812 10.6875 0.0359536 0.0454844 2022-03-10T01:37:14.173 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 329 16 115636 115620 21.9609 15.3125 0.106371 0.0455243 2022-03-10T01:37:14.673 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:14.680722 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:37:14.717 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:14.726195 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:37:15.173 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 330 16 115826 115810 21.9304 11.875 0.0844259 0.0455874 2022-03-10T01:37:16.173 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 331 16 116035 116019 21.9036 13.0625 0.260262 0.0456326 2022-03-10T01:37:17.173 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 332 16 116318 116302 21.8909 17.6875 0.0653083 0.0456713 2022-03-10T01:37:17.361 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:17.370200 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:37:18.173 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 333 16 116534 116518 21.8657 13.5 0.0235244 0.0457251 2022-03-10T01:37:19.173 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 334 16 116803 116787 21.8505 16.8125 0.213627 0.0457548 2022-03-10T01:37:20.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 335 16 117091 117075 21.839 18 0.0451467 0.0457802 2022-03-10T01:37:20.324 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:20.332320 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:37:20.324 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:20.332345 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:37:21.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:21.162778 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:37:21.173 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 336 16 117332 117316 21.8189 15.0625 0.0295452 0.0458219 2022-03-10T01:37:22.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 337 16 117560 117544 21.7964 14.25 0.0995509 0.0458624 2022-03-10T01:37:23.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 338 16 117790 117774 21.7745 14.375 0.0245702 0.0459065 2022-03-10T01:37:24.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 339 16 118046 118030 21.7574 16 0.0663334 0.0459508 2022-03-10T01:37:25.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:2022-03-10T01:37:25.183556+0000 min lat: 0.00129056 max lat: 10.3569 avg lat: 0.0459733 2022-03-10T01:37:25.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-10T01:37:25.175 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 340 16 118338 118322 21.7471 18.25 0.0244298 0.0459733 2022-03-10T01:37:26.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 341 16 118614 118598 21.7339 17.25 0.033891 0.0460031 2022-03-10T01:37:26.355 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:26.361514 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:37:26.444 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:26.449516 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:37:27.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 342 16 118875 118859 21.718 16.3125 0.0733707 0.0460344 2022-03-10T01:37:28.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 343 16 119158 119142 21.7063 17.6875 0.0443576 0.0460588 2022-03-10T01:37:29.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 344 16 119422 119406 21.6912 16.5 0.0361259 0.0460914 2022-03-10T01:37:29.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:29.430653 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:37:30.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 345 16 119691 119675 21.677 16.8125 0.0171008 0.046114 2022-03-10T01:37:31.174 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 346 16 119935 119919 21.6584 15.25 0.0696981 0.0461636 2022-03-10T01:37:31.292 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:31.301373 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:37:31.293 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:31.301390 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:37:32.089 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:32.095842 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:37:32.176 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 347 16 120157 120141 21.6359 13.875 0.0737734 0.0462083 2022-03-10T01:37:33.175 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 348 16 120409 120393 21.619 15.75 0.0157037 0.0462455 2022-03-10T01:37:34.175 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 349 16 120600 120584 21.5913 11.9375 0.0229815 0.046303 2022-03-10T01:37:35.176 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 350 16 120877 120861 21.579 17.3125 0.0299841 0.0463328 2022-03-10T01:37:36.176 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 351 16 121186 121170 21.5726 19.3125 0.00468394 0.046341 2022-03-10T01:37:37.176 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 352 16 121449 121433 21.558 16.4375 0.131001 0.0463743 2022-03-10T01:37:37.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:37.461924 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:37:37.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:37.629339 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:37:38.176 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 353 16 121770 121754 21.5538 20.0625 0.056057 0.0463857 2022-03-10T01:37:39.176 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 354 16 122050 122034 21.5423 17.5 0.0259872 0.0464089 2022-03-10T01:37:39.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:39.802721 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:37:39.878 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:39.883938 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:37:39.878 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:39.883964 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:37:40.176 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 355 16 122345 122329 21.5335 18.4375 0.0172261 0.0464301 2022-03-10T01:37:40.668 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:40.673030 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:37:41.176 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 356 16 122628 122612 21.5227 17.6875 0.0380513 0.0464537 2022-03-10T01:37:42.176 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 357 16 122856 122840 21.5024 14.25 0.0319426 0.0464937 2022-03-10T01:37:43.177 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 358 16 123165 123149 21.4962 19.3125 0.00858486 0.0465087 2022-03-10T01:37:43.812 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:43.817335 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:37:44.177 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:44.185204 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:37:44.178 INFO:tasks.radosbench.radosbench.a.smithi186.stdout: 359 16 123339 123323 21.4667 10.875 0.0318864 0.0465725 2022-03-10T01:37:45.178 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Total time run: 360.033 2022-03-10T01:37:45.178 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Total writes made: 123608 2022-03-10T01:37:45.179 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Write size: 65536 2022-03-10T01:37:45.179 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Object size: 65536 2022-03-10T01:37:45.179 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Bandwidth (MB/sec): 21.4578 2022-03-10T01:37:45.179 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Stddev Bandwidth: 16.7814 2022-03-10T01:37:45.179 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Max bandwidth (MB/sec): 79.5 2022-03-10T01:37:45.180 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Min bandwidth (MB/sec): 0 2022-03-10T01:37:45.180 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Average IOPS: 343 2022-03-10T01:37:45.180 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Stddev IOPS: 268.511 2022-03-10T01:37:45.180 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Max IOPS: 1272 2022-03-10T01:37:45.181 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Min IOPS: 0 2022-03-10T01:37:45.181 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Average Latency(s): 0.0465958 2022-03-10T01:37:45.181 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Stddev Latency(s): 0.246835 2022-03-10T01:37:45.181 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Max latency(s): 10.3569 2022-03-10T01:37:45.181 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Min latency(s): 0.00129056 2022-03-10T01:37:45.182 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Cleaning up (deleting benchmark objects) 2022-03-10T01:37:46.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:46.183853 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:37:46.348 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:46.356967 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:37:46.348 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:46.356992 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:37:47.008 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:47.015245 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:37:50.537 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:50.544528 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:37:51.182 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:51.190693 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:37:53.336 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:53.344733 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:37:53.460 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:53.453247 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:37:53.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:53.453276 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:37:54.069 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:54.073717 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:37:57.279 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:57.287827 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:37:58.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:37:58.581548 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:38:00.876 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:00.885152 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:38:01.042 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:01.044888 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:38:01.043 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:01.050953 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:38:01.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:01.590268 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:38:04.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:04.431719 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:38:07.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:07.161684 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:38:09.890 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:09.899177 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:38:12.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:12.939853 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:38:12.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:12.939880 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:38:13.519 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:13.525973 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:38:17.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:17.739991 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:38:22.862 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:22.871087 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:38:24.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:24.829415 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:38:24.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:24.950611 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:38:24.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:24.950638 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:38:25.499 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:25.508713 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:38:28.472 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:28.480075 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:38:38.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:38.729709 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:38:40.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:40.811096 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:38:40.891 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:40.900269 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:38:40.892 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:40.900287 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:38:41.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:41.599491 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:38:44.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:38:44.462914 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:39:04.941 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:04.950224 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:39:07.192 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:07.201354 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:39:07.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:07.329917 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:39:07.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:07.329947 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:39:07.940 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:07.945057 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:39:11.516 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:11.525869 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:39:52.513 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:52.521986 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:39:54.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:54.446766 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:39:54.571 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:54.580719 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:39:54.572 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:54.580746 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:39:55.211 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:55.220050 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:39:59.387 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:39:59.394886 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:41:21.318 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:41:21.325155 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:41:23.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:41:23.152841 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:41:23.288 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:41:23.297184 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:41:23.289 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:41:23.297211 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:41:24.014 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:41:24.021737 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:41:27.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:41:27.154003 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:44:11.009 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:44:11.017133 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:44:12.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:44:12.765160 I | ceph-object-controller: reconciling object store deployments 2022-03-10T01:44:12.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:44:12.877457 I | ceph-object-controller: ceph object store gateway service running at 10.96.148.7 2022-03-10T01:44:12.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:44:12.877485 I | ceph-object-controller: reconciling object store pools 2022-03-10T01:44:13.618 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:44:13.628650 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T01:44:17.460 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:44:17.469121 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-10T01:45:07.923 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Removed 123608 objects 2022-03-10T01:45:07.930 INFO:tasks.radosbench.radosbench.a.smithi186.stdout:Clean up completed and total clean up time :442.742 2022-03-10T01:45:08.233 INFO:tasks.rook.ceph_manager.ceph:removing pool_name unique_pool_0 2022-03-10T01:45:08.235 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2022-03-10T01:45:11.882 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'unique_pool_0' removed 2022-03-10T01:45:11.901 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-03-10T01:45:11.919 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-03-10T01:45:11.920 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-03-10T01:45:13.056 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-10T01:45:13.070 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:45:15.244 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:45:15.245 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:36:18.000000Z", "last_refresh": "2022-03-10T01:45:13.810985Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:45:13.810985Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:45:13.810985Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:45:13.810985Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:45:13.810985Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:45:13.810985Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-10T01:45:13.810985Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:45:13.810985Z", "running": 0, "size": 1}}] 2022-03-10T01:45:15.263 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch rm mds.foo 2022-03-10T01:45:16.169 INFO:teuthology.orchestra.run.smithi186.stdout:Removed cephfilesystems/foo 2022-03-10T01:45:16.191 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch rm rbd-mirror 2022-03-10T01:45:16.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:16.254159 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:45:16.724 INFO:teuthology.orchestra.run.smithi186.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-10T01:45:16.754 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch rm rgw.foo 2022-03-10T01:45:16.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:16.767672 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-10T01:45:17.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:17.589989 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-10T01:45:17.943 INFO:teuthology.orchestra.run.smithi186.stdout:Removed cephobjectstores/foo 2022-03-10T01:45:17.964 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:45:17.966 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:17.977044 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:45:20.045 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:45:20.046 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:36:18.000000Z", "last_refresh": "2022-03-10T01:45:18.839724Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:45:18.839724Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:45:18.839724Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:45:18.839724Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:45:18.839724Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:45:18.839724Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:45:18.839724Z", "running": 0, "size": 1}}] 2022-03-10T01:45:20.078 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:20.087887 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-10T01:45:20.078 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:20.087908 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-10T01:45:20.079 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:20.087913 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-10T01:45:20.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:20.856420 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-10T01:45:21.460 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:21.470977 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-10T01:45:22.030 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:22.040549 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-10T01:45:22.031 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:22.040569 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-10T01:45:22.031 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:22.040573 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-10T01:45:23.053 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:23.063813 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T01:45:23.053+0000 7f04b3e49340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T01:45:23.054 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:45:23.053+0000 7f04b3e49340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T01:45:23.054 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:45:23.053+0000 7f04b3e49340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T01:45:23.054 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T01:45:23.054 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:23.063833 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T01:45:23.053+0000 7f04b3e49340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T01:45:23.055 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:45:23.053+0000 7f04b3e49340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T01:45:23.055 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:45:23.053+0000 7f04b3e49340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T01:45:23.055 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T01:45:23.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:23.291618 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-10T01:45:23.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:23.291653 I | ceph-file-controller: downing filesystem "foo" 2022-03-10T01:45:25.222 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:25.232055 I | ceph-file-controller: downed filesystem "foo" 2022-03-10T01:45:28.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:28.855119 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-10T01:45:28.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:28.855143 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-10T01:45:30.068 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:45:30.825 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:45:30.826 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:36:18.000000Z", "last_refresh": "2022-03-10T01:45:30.696774Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:45:30.696774Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:45:30.696774Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:45:30.696774Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:45:30.696774Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:45:30.696774Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:45:30.696774Z", "running": 0, "size": 1}}] 2022-03-10T01:45:32.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:32.014907 I | cephclient: purge completed for pool "foo-metadata" 2022-03-10T01:45:32.418 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:32.428865 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-10T01:45:32.419 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:32.428883 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-10T01:45:34.387 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:34.398106 I | cephclient: purge completed for pool "foo-data0" 2022-03-10T01:45:34.388 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:34.398136 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-10T01:45:34.451 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:34.461777 E | ceph-file-controller: failed to reconcile failed to remove finalizer: failed to remove finalizer "cephfilesystem.ceph.rook.io" on "foo": Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:45:34.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:34.472328 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:45:35.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:35.292644 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-10T01:45:35.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:35.443475 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-10T01:45:35.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:35.466097 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-10T01:45:35.473 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:35.483569 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-10T01:45:35.473 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:35.483771 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-10T01:45:39.638 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:45:39.649034 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi186": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:45:40.848 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:45:41.960 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:45:41.960 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:36:18.000000Z", "last_refresh": "2022-03-10T01:45:41.487631Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:45:41.487631Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:45:41.487631Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:45:41.487631Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:45:41.487631Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:45:41.487631Z", "running": 0, "size": 1}}] 2022-03-10T01:45:51.981 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:45:52.940 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:45:52.940 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:36:18.000000Z", "last_refresh": "2022-03-10T01:45:52.843746Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:45:52.843746Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:45:52.843746Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:45:52.843746Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:45:52.843746Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:45:52.843746Z", "running": 0, "size": 1}}] 2022-03-10T01:46:02.965 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:46:03.929 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:46:03.929 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:46:03.818199Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:46:03.818199Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:46:03.818199Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:46:03.818199Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:46:03.818199Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:46:03.818199Z", "running": 0, "size": 1}}] 2022-03-10T01:46:13.953 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:46:17.488 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:46:17.489 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:46:17.411412Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:46:17.411412Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:46:17.411412Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:46:17.411412Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:46:17.411412Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:46:17.411412Z", "running": 0, "size": 1}}] 2022-03-10T01:46:27.513 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:46:28.240 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:46:28.240 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:46:28.173512Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:46:28.173512Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:46:28.173512Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:46:28.173512Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:46:28.173512Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:46:28.173512Z", "running": 0, "size": 1}}] 2022-03-10T01:46:38.262 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:46:38.949 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:46:38.950 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:46:38.864735Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:46:38.864735Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:46:38.864735Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:46:38.864735Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:46:38.864735Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:46:38.864735Z", "running": 0, "size": 1}}] 2022-03-10T01:46:48.974 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:46:49.629 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:46:49.630 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:46:49.518560Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:46:49.518560Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:46:49.518560Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:46:49.518560Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:46:49.518560Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:46:49.518560Z", "running": 0, "size": 1}}] 2022-03-10T01:46:59.651 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:47:01.338 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:47:01.338 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:47:00.221927Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:47:00.221927Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:47:00.221927Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:47:00.221927Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:47:00.221927Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:47:00.221927Z", "running": 0, "size": 1}}] 2022-03-10T01:47:11.364 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:47:11.957 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:47:11.957 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:47:11.881828Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:47:11.881828Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:47:11.881828Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:47:11.881828Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:47:11.881828Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:47:11.881828Z", "running": 0, "size": 1}}] 2022-03-10T01:47:21.979 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:47:22.645 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:47:22.646 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:47:22.532967Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:47:22.532967Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:47:22.532967Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:47:22.532967Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:47:22.532967Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:47:22.532967Z", "running": 0, "size": 1}}] 2022-03-10T01:47:32.669 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:47:33.357 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:47:33.357 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:47:33.200131Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:47:33.200131Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:47:33.200131Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:47:33.200131Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:47:33.200131Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:47:33.200131Z", "running": 0, "size": 1}}] 2022-03-10T01:47:43.382 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:47:44.148 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:47:44.148 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:47:44.069892Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:47:44.069892Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:47:44.069892Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:47:44.069892Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:47:44.069892Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:47:44.069892Z", "running": 0, "size": 1}}] 2022-03-10T01:47:54.172 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:47:54.790 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:47:54.790 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:47:54.694080Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:47:54.694080Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:47:54.694080Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:47:54.694080Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:47:54.694080Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:47:54.694080Z", "running": 0, "size": 1}}] 2022-03-10T01:48:04.816 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:48:05.444 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:48:05.445 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:48:05.364403Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:48:05.364403Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:48:05.364403Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:48:05.364403Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:48:05.364403Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:48:05.364403Z", "running": 0, "size": 1}}] 2022-03-10T01:48:15.470 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:48:16.173 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:48:16.174 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:48:16.016833Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:48:16.016833Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:48:16.016833Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:48:16.016833Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:48:16.016833Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:48:16.016833Z", "running": 0, "size": 1}}] 2022-03-10T01:48:26.200 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:48:26.867 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:48:26.867 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:48:26.792717Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:48:26.792717Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:48:26.792717Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:48:26.792717Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:48:26.792717Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:48:26.792717Z", "running": 0, "size": 1}}] 2022-03-10T01:48:36.897 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:48:37.766 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:48:37.766 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:48:37.387714Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:48:37.387714Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:48:37.387714Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:48:37.387714Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:48:37.387714Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:48:37.387714Z", "running": 0, "size": 1}}] 2022-03-10T01:48:47.786 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:48:48.447 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:48:48.448 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:48:48.363361Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:48:48.363361Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:48:48.363361Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:48:48.363361Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:48:48.363361Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:48:48.363361Z", "running": 0, "size": 1}}] 2022-03-10T01:48:58.468 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:48:59.120 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:48:59.120 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:48:59.036969Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:48:59.036969Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:48:59.036969Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:48:59.036969Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:48:59.036969Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:48:59.036969Z", "running": 0, "size": 1}}] 2022-03-10T01:49:09.145 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:49:09.760 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:49:09.760 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:49:09.675639Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:49:09.675639Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:49:09.675639Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:49:09.675639Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:49:09.675639Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:49:09.675639Z", "running": 0, "size": 1}}] 2022-03-10T01:49:19.784 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:49:20.418 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:49:20.419 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:49:20.307452Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:49:20.307452Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:49:20.307452Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:49:20.307452Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:49:20.307452Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:49:20.307452Z", "running": 0, "size": 1}}] 2022-03-10T01:49:30.443 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:49:31.043 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:49:31.044 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:49:30.962434Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:49:30.962434Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:49:30.962434Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:49:30.962434Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:49:30.962434Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:49:30.962434Z", "running": 0, "size": 1}}] 2022-03-10T01:49:41.068 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:49:41.742 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:49:41.742 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:49:41.581988Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:49:41.581988Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:49:41.581988Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:49:41.581988Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:49:41.581988Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:49:41.581988Z", "running": 0, "size": 1}}] 2022-03-10T01:49:45.146 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:49:45.158216 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T01:49:45.762 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:49:45.773460 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T01:49:45.761+0000 7f9fa2e2d340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T01:49:45.763 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:49:45.761+0000 7f9fa2e2d340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T01:49:45.763 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:49:45.761+0000 7f9fa2e2d340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T01:49:45.763 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T01:49:45.763 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 01:49:45.773482 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T01:49:45.761+0000 7f9fa2e2d340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T01:49:45.764 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:49:45.761+0000 7f9fa2e2d340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T01:49:45.764 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T01:49:45.761+0000 7f9fa2e2d340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T01:49:45.764 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T01:49:51.766 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:49:52.439 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:49:52.439 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:49:52.320758Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:49:52.320758Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:49:52.320758Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:49:52.320758Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:49:52.320758Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:49:52.320758Z", "running": 0, "size": 1}}] 2022-03-10T01:50:02.463 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:50:03.086 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:50:03.086 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:50:02.999016Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:50:02.999016Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:50:02.999016Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:50:02.999016Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:50:02.999016Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:50:02.999016Z", "running": 0, "size": 1}}] 2022-03-10T01:50:13.111 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:50:14.084 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:50:14.084 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:50:13.999311Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:50:13.999311Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:50:13.999311Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:50:13.999311Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:50:13.999311Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:50:13.999311Z", "running": 0, "size": 1}}] 2022-03-10T01:50:24.106 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:50:24.714 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:50:24.714 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:50:24.635086Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:50:24.635086Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:50:24.635086Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:50:24.635086Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:50:24.635086Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:50:24.635086Z", "running": 0, "size": 1}}] 2022-03-10T01:50:34.739 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:50:35.384 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:50:35.385 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:50:35.245235Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:50:35.245235Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:50:35.245235Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:50:35.245235Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:50:35.245235Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:50:35.245235Z", "running": 0, "size": 1}}] 2022-03-10T01:50:45.409 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:50:46.023 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:50:46.023 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:50:45.944623Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:50:45.944623Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:50:45.944623Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:50:45.944623Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:50:45.944623Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:50:45.944623Z", "running": 0, "size": 1}}] 2022-03-10T01:50:56.047 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:50:56.700 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:50:56.700 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:50:56.555388Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:50:56.555388Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:50:56.555388Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:50:56.555388Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:50:56.555388Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:50:56.555388Z", "running": 0, "size": 1}}] 2022-03-10T01:51:06.723 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:51:07.354 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:51:07.354 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:51:07.258301Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:51:07.258301Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:51:07.258301Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:51:07.258301Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:51:07.258301Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:51:07.258301Z", "running": 0, "size": 1}}] 2022-03-10T01:51:17.382 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:51:18.046 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:51:18.046 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:51:17.944671Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:51:17.944671Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:51:17.944671Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:51:17.944671Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:51:17.944671Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:51:17.944671Z", "running": 0, "size": 1}}] 2022-03-10T01:51:28.069 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:51:28.663 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:51:28.663 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:51:28.587391Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:51:28.587391Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:51:28.587391Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:51:28.587391Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:51:28.587391Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:51:28.587391Z", "running": 0, "size": 1}}] 2022-03-10T01:51:38.687 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:51:39.316 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:51:39.319 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:51:39.227657Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:51:39.227657Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:51:39.227657Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:51:39.227657Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:51:39.227657Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:51:39.227657Z", "running": 0, "size": 1}}] 2022-03-10T01:51:49.337 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:51:49.967 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:51:49.967 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:51:49.885372Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:51:49.885372Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:51:49.885372Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:51:49.885372Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:51:49.885372Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:51:49.885372Z", "running": 0, "size": 1}}] 2022-03-10T01:51:59.991 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:52:00.664 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:52:00.664 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:52:00.519966Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:52:00.519966Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:52:00.519966Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:52:00.519966Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:52:00.519966Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:52:00.519966Z", "running": 0, "size": 1}}] 2022-03-10T01:52:10.686 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:52:11.309 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:52:11.310 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:52:11.228723Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:52:11.228723Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:52:11.228723Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:52:11.228723Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:52:11.228723Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:52:11.228723Z", "running": 0, "size": 1}}] 2022-03-10T01:52:21.330 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:52:22.070 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:52:22.070 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:52:21.883954Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:52:21.883954Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:52:21.883954Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:52:21.883954Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:52:21.883954Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:52:21.883954Z", "running": 0, "size": 1}}] 2022-03-10T01:52:32.096 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:52:32.717 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:52:32.718 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:52:32.617961Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:52:32.617961Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:52:32.617961Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:52:32.617961Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:52:32.617961Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:52:32.617961Z", "running": 0, "size": 1}}] 2022-03-10T01:52:42.743 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:52:43.344 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:52:43.344 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:52:43.259520Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:52:43.259520Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:52:43.259520Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:52:43.259520Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:52:43.259520Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:52:43.259520Z", "running": 0, "size": 1}}] 2022-03-10T01:52:53.370 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:52:54.012 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:52:54.013 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:52:53.899359Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:52:53.899359Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:52:53.899359Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:52:53.899359Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:52:53.899359Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:52:53.899359Z", "running": 0, "size": 1}}] 2022-03-10T01:53:04.034 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:53:04.668 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:53:04.669 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:53:04.584259Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:53:04.584259Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:53:04.584259Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:53:04.584259Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:53:04.584259Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:53:04.584259Z", "running": 0, "size": 1}}] 2022-03-10T01:53:14.695 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:53:15.365 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:53:15.366 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:53:15.276092Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:53:15.276092Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:53:15.276092Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:53:15.276092Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:53:15.276092Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:53:15.276092Z", "running": 0, "size": 1}}] 2022-03-10T01:53:25.390 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:53:26.084 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:53:26.085 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:53:25.999980Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:53:25.999980Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:53:25.999980Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:53:25.999980Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:53:25.999980Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:53:25.999980Z", "running": 0, "size": 1}}] 2022-03-10T01:53:36.110 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:53:36.766 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:53:36.766 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:53:36.680074Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:53:36.680074Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:53:36.680074Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:53:36.680074Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:53:36.680074Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:53:36.680074Z", "running": 0, "size": 1}}] 2022-03-10T01:53:46.791 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:53:47.453 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:53:47.454 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:53:47.369742Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:53:47.369742Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:53:47.369742Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:53:47.369742Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:53:47.369742Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:53:47.369742Z", "running": 0, "size": 1}}] 2022-03-10T01:53:57.478 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:53:58.106 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:53:58.107 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:53:57.997541Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:53:57.997541Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:53:57.997541Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:53:57.997541Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:53:57.997541Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:53:57.997541Z", "running": 0, "size": 1}}] 2022-03-10T01:54:08.134 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:54:08.790 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:54:08.790 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:54:08.658121Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:54:08.658121Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:54:08.658121Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:54:08.658121Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:54:08.658121Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:54:08.658121Z", "running": 0, "size": 1}}] 2022-03-10T01:54:18.818 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:54:19.455 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:54:19.456 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:54:19.362860Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:54:19.362860Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:54:19.362860Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:54:19.362860Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:54:19.362860Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:54:19.362860Z", "running": 0, "size": 1}}] 2022-03-10T01:54:29.511 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:54:30.115 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:54:30.116 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:54:29.984971Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:54:29.984971Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:54:29.984971Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:54:29.984971Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:54:29.984971Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:54:29.984971Z", "running": 0, "size": 1}}] 2022-03-10T01:54:40.140 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:54:40.750 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:54:40.751 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:54:40.658702Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:54:40.658702Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:54:40.658702Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:54:40.658702Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:54:40.658702Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:54:40.658702Z", "running": 0, "size": 1}}] 2022-03-10T01:54:50.777 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:54:51.399 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:54:51.399 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:54:51.316030Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:54:51.316030Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:54:51.316030Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:54:51.316030Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:54:51.316030Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:54:51.316030Z", "running": 0, "size": 1}}] 2022-03-10T01:55:01.423 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:55:02.097 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:55:02.098 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:55:02.011152Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:55:02.011152Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:55:02.011152Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:55:02.011152Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:55:02.011152Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:55:02.011152Z", "running": 0, "size": 1}}] 2022-03-10T01:55:12.121 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:55:12.723 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:55:12.724 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:55:12.634415Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:55:12.634415Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:55:12.634415Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:55:12.634415Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:55:12.634415Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:55:12.634415Z", "running": 0, "size": 1}}] 2022-03-10T01:55:22.749 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:55:23.389 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:55:23.390 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:55:23.297998Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:55:23.297998Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:55:23.297998Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:55:23.297998Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:55:23.297998Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:55:23.297998Z", "running": 0, "size": 1}}] 2022-03-10T01:55:33.413 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:55:34.027 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:55:34.028 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:55:33.942219Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:55:33.942219Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:55:33.942219Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:55:33.942219Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:55:33.942219Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:55:33.942219Z", "running": 0, "size": 1}}] 2022-03-10T01:55:44.050 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:55:44.657 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:55:44.658 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:55:44.569804Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:55:44.569804Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:55:44.569804Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:55:44.569804Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:55:44.569804Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:55:44.569804Z", "running": 0, "size": 1}}] 2022-03-10T01:55:54.682 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:55:55.319 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:55:55.320 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:55:55.236790Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:55:55.236790Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:55:55.236790Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:55:55.236790Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:55:55.236790Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:55:55.236790Z", "running": 0, "size": 1}}] 2022-03-10T01:56:05.343 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:56:06.333 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:56:06.333 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:56:06.194922Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:56:06.194922Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:56:06.194922Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:56:06.194922Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:56:06.194922Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:56:06.194922Z", "running": 0, "size": 1}}] 2022-03-10T01:56:16.357 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:56:20.470 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:56:20.470 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:56:20.383028Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:56:20.383028Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:56:20.383028Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:56:20.383028Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:56:20.383028Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:56:20.383028Z", "running": 0, "size": 1}}] 2022-03-10T01:56:30.494 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:56:31.096 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:56:31.096 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:56:31.003590Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:56:31.003590Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:56:31.003590Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:56:31.003590Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:56:31.003590Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:56:31.003590Z", "running": 0, "size": 1}}] 2022-03-10T01:56:41.116 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:56:41.851 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:56:41.852 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:56:41.675448Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:56:41.675448Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:56:41.675448Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:56:41.675448Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:56:41.675448Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:56:41.675448Z", "running": 0, "size": 1}}] 2022-03-10T01:56:51.873 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:56:52.485 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:56:52.485 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:56:52.395529Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:56:52.395529Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:56:52.395529Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:56:52.395529Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:56:52.395529Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:56:52.395529Z", "running": 0, "size": 1}}] 2022-03-10T01:57:02.508 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:57:03.136 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:57:03.137 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:57:02.990759Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:57:02.990759Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:57:02.990759Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:57:02.990759Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:57:02.990759Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:57:02.990759Z", "running": 0, "size": 1}}] 2022-03-10T01:57:13.159 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:57:13.779 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:57:13.779 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:57:13.680024Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:57:13.680024Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:57:13.680024Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:57:13.680024Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:57:13.680024Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:57:13.680024Z", "running": 0, "size": 1}}] 2022-03-10T01:57:23.803 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:57:24.410 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:57:24.411 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:57:24.318724Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:57:24.318724Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:57:24.318724Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:57:24.318724Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:57:24.318724Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:57:24.318724Z", "running": 0, "size": 1}}] 2022-03-10T01:57:34.432 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:57:35.062 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:57:35.062 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:57:34.951608Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:57:34.951608Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:57:34.951608Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:57:34.951608Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:57:34.951608Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:57:34.951608Z", "running": 0, "size": 1}}] 2022-03-10T01:57:45.086 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:57:45.723 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:57:45.723 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:57:45.638246Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:57:45.638246Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:57:45.638246Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:57:45.638246Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:57:45.638246Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:57:45.638246Z", "running": 0, "size": 1}}] 2022-03-10T01:57:55.746 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:57:56.363 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:57:56.363 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:57:56.272575Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:57:56.272575Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:57:56.272575Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:57:56.272575Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:57:56.272575Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:57:56.272575Z", "running": 0, "size": 1}}] 2022-03-10T01:58:06.388 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:58:07.042 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:58:07.043 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:58:06.944149Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:58:06.944149Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:58:06.944149Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:58:06.944149Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:58:06.944149Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:58:06.944149Z", "running": 0, "size": 1}}] 2022-03-10T01:58:17.063 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:58:17.709 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:58:17.709 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:58:17.558894Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:58:17.558894Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:58:17.558894Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:58:17.558894Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:58:17.558894Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:58:17.558894Z", "running": 0, "size": 1}}] 2022-03-10T01:58:27.733 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:58:28.395 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:58:28.395 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:58:28.298883Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:58:28.298883Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:58:28.298883Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:58:28.298883Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:58:28.298883Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:58:28.298883Z", "running": 0, "size": 1}}] 2022-03-10T01:58:38.420 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:58:39.048 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:58:39.048 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:58:38.954652Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:58:38.954652Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:58:38.954652Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:58:38.954652Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:58:38.954652Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:58:38.954652Z", "running": 0, "size": 1}}] 2022-03-10T01:58:49.072 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:58:49.723 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:58:49.723 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:58:49.623441Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:58:49.623441Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:58:49.623441Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:58:49.623441Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:58:49.623441Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:58:49.623441Z", "running": 0, "size": 1}}] 2022-03-10T01:58:59.744 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:59:00.388 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:59:00.388 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:59:00.299770Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:59:00.299770Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:59:00.299770Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:59:00.299770Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:59:00.299770Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:59:00.299770Z", "running": 0, "size": 1}}] 2022-03-10T01:59:10.409 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:59:11.021 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:59:11.021 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:59:10.928384Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:59:10.928384Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:59:10.928384Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:59:10.928384Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:59:10.928384Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:59:10.928384Z", "running": 0, "size": 1}}] 2022-03-10T01:59:21.045 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:59:21.665 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:59:21.666 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:59:21.573668Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:59:21.573668Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:59:21.573668Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:59:21.573668Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:59:21.573668Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:59:21.573668Z", "running": 0, "size": 1}}] 2022-03-10T01:59:31.689 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:59:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:59:32.431 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:59:32.310197Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:59:32.310197Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:59:32.310197Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:59:32.310197Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:59:32.310197Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:59:32.310197Z", "running": 0, "size": 1}}] 2022-03-10T01:59:42.453 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:59:43.072 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:59:43.072 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:59:42.977482Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:59:42.977482Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:59:42.977482Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:59:42.977482Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:59:42.977482Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:59:42.977482Z", "running": 0, "size": 1}}] 2022-03-10T01:59:53.096 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T01:59:53.727 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T01:59:53.727 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T01:59:53.638438Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T01:59:53.638438Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T01:59:53.638438Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T01:59:53.638438Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T01:59:53.638438Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T01:59:53.638438Z", "running": 0, "size": 1}}] 2022-03-10T02:00:03.770 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T02:00:04.418 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T02:00:04.419 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T02:00:04.327332Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T02:00:04.327332Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T02:00:04.327332Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T02:00:04.327332Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:00:04.327332Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:00:04.327332Z", "running": 0, "size": 1}}] 2022-03-10T02:00:14.442 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T02:00:15.005 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T02:00:15.005 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T02:00:14.925421Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T02:00:14.925421Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T02:00:14.925421Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T02:00:14.925421Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:00:14.925421Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:00:14.925421Z", "running": 0, "size": 1}}] 2022-03-10T02:00:25.031 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T02:00:25.677 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T02:00:25.677 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T02:00:25.545616Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T02:00:25.545616Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T02:00:25.545616Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T02:00:25.545616Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:00:25.545616Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:00:25.545616Z", "running": 0, "size": 1}}] 2022-03-10T02:00:35.699 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T02:00:36.330 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T02:00:36.330 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T02:00:36.238284Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T02:00:36.238284Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T02:00:36.238284Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T02:00:36.238284Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:00:36.238284Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:00:36.238284Z", "running": 0, "size": 1}}] 2022-03-10T02:00:46.350 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T02:00:46.997 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T02:00:46.998 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T02:00:46.894409Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T02:00:46.894409Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T02:00:46.894409Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T02:00:46.894409Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:00:46.894409Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:00:46.894409Z", "running": 0, "size": 1}}] 2022-03-10T02:00:57.039 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T02:00:57.668 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T02:00:57.668 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T02:00:57.569576Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T02:00:57.569576Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T02:00:57.569576Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T02:00:57.569576Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:00:57.569576Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:00:57.569576Z", "running": 0, "size": 1}}] 2022-03-10T02:01:07.693 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T02:01:08.379 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T02:01:08.379 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T02:01:08.241160Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T02:01:08.241160Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T02:01:08.241160Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T02:01:08.241160Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:01:08.241160Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:01:08.241160Z", "running": 0, "size": 1}}] 2022-03-10T02:01:18.404 DEBUG:teuthology.orchestra.run.smithi186:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-7sxwg -- ceph orch ls -f json 2022-03-10T02:01:19.048 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-03-10T02:01:19.049 INFO:teuthology.orchestra.run.smithi186.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:45:39.000000Z", "last_refresh": "2022-03-10T02:01:18.958835Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:29.000000Z", "last_refresh": "2022-03-10T02:01:18.958835Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:26:03.000000Z", "last_refresh": "2022-03-10T02:01:18.958835Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:28:28.000000Z", "last_refresh": "2022-03-10T02:01:18.958835Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:01:18.958835Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:01:18.958835Z", "running": 0, "size": 1}}] 2022-03-10T02:01:19.070 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:01:19.071 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:01:19.071 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-10T02:01:19.072 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T02:01:19.085 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:01:19.086 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:01:19.121 DEBUG:teuthology.orchestra.remote:smithi186:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-10T02:01:19.136 DEBUG:teuthology.orchestra.run.smithi186:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-10T02:01:19.220 INFO:teuthology.orchestra.run.smithi186.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-10T02:01:19.249 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:01:19.251 DEBUG:teuthology.orchestra.run.smithi186:> kubectl delete -f cluster.yaml 2022-03-10T02:01:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-10T02:01:19.376 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:19.379053 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-10T02:01:19.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:19.622757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:01:19.629 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:19.631842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:01:19.632 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:19.633598 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:01:29.854 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:29.855933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:01:29.866 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:29.868743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:01:40.091 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:40.093058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:01:40.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:40.106348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:01:50.319 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:50.320601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:01:50.333 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:01:50.333402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:00.550 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:00.551922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:01.008 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:00.569978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:10.815 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:10.817223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:10.830 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:10.832647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:21.049 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:21.050912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:21.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:21.065615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:31.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:31.282389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:31.454 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:31.296432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:41.514 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:41.514639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:41.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:41.526935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:51.747 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:51.748527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:02:51.758 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:02:51.760084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:01.987 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:01.988199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:02.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:02.005577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:12.223 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:12.224113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:12.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:12.236648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:22.454 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:22.454950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:22.467 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:22.468941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:32.687 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:32.688513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:32.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:32.699554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:42.961 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:42.962500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:42.973 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:42.974123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:53.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:53.192140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:03:53.205 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:03:53.205605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:03.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:03.426024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:03.437 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:03.437831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:13.661 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:13.659292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:13.675 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:13.676126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:23.888 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:23.888942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:23.899 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:23.900266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:34.117 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:34.117313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:34.127 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:34.128437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:34.192 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:34.193338 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 02:04:34.185750166 +0000 UTC m=+2402.324289098 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:04:44.347 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:44.347139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:44.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:44.358327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:54.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:54.582803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:04:54.593 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:04:54.594446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:04.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:04.809169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:04.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:04.823598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:15.040 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:15.040521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:15.051 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:15.052218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:25.272 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:25.272571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:25.283 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:25.283697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:35.505 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:35.505571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:35.518 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:35.518259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:45.737 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:45.737992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:45.749 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:45.749830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:56.031 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:56.031785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:05:56.045 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:05:56.045620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:06.268 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:06.268431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:06.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:06.280877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:16.540 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:16.540471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:16.552 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:16.552150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:25.781 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:25.781678 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T02:06:26.423 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:26.422652 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:06:26.416+0000 7f84cc25b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:06:26.424 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:06:26.416+0000 7f84cc25b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:06:26.424 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:06:26.416+0000 7f84cc25b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:06:26.424 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:06:26.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:26.422686 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:06:26.416+0000 7f84cc25b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:06:26.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:06:26.416+0000 7f84cc25b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:06:26.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:06:26.416+0000 7f84cc25b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:06:26.426 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:06:26.798 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:26.798914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:26.811 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:26.811092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:37.031 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:37.029842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:37.040 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:37.040657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:47.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:47.262758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:47.278 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:47.278908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:57.495 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:57.495313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:06:57.506 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:06:57.506684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:07.730 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:07.730700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:07.743 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:07.743221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:17.964 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:17.964262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:17.977 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:17.976881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:28.200 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:28.200024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:28.210 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:28.210266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:38.469 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:38.469219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:38.519 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:38.519891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:48.740 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:48.740365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:48.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:48.794372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:59.015 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:59.014380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:07:59.026 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:07:59.025880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:09.243 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:09.243393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:09.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:09.256371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:19.475 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:19.475238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:19.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:19.485602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:29.703 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:29.702444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:29.713 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:29.713173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:39.960 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:39.960275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:39.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:39.972415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:50.243 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:50.242794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:08:50.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:08:50.255078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:00.474 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:00.473369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:00.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:00.484726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:10.765 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:10.764237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:10.775 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:10.775383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:20.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:20.993675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:21.005 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:21.004875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:31.229 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:31.227968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:31.242 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:31.241672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:41.463 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:41.462601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:41.477 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:41.475365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:51.696 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:51.695311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:09:51.786 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:09:51.785599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:02.007 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:02.006337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:02.018 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:02.017892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:12.278 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:12.277648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:12.293 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:12.293108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:22.512 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:22.511124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:22.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:22.524706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:32.741 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:32.740690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:32.753 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:32.752476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:43.033 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:43.032160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:43.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:43.045311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:53.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:53.265741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:10:53.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:10:53.280782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:03.500 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:03.499613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:03.513 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:03.512292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:13.800 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:13.799995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:13.811 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:13.810772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:24.053 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:24.052555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:24.062 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:24.061406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:34.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:34.280352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:34.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:34.290634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:44.824 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:44.823624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:44.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:44.832630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:55.092 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:55.091109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:11:55.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:11:55.103104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:05.410 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:05.408220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:05.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:05.420669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:16.292 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:16.290841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:16.301 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:16.300778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:26.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:26.552121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:26.567 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:26.565784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:36.875 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:36.873794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:36.887 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:36.886758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:47.284 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:47.282780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:47.294 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:47.293728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:57.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:57.522632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:12:57.538 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:12:57.537490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:07.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:07.756487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:07.772 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:07.771194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:18.065 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:18.064282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:18.111 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:18.110034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:28.331 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:28.330360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:28.341 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:28.339999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:38.564 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:38.562086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:38.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:38.573825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:48.836 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:48.833728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:48.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:48.844350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:59.073 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:59.072055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:13:59.085 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:13:59.084229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:09.307 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:09.305966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:09.342 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:09.340284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:19.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:19.696193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:19.712 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:19.711496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:29.931 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:29.929780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:29.944 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:29.943261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:40.165 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:40.163063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:40.178 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:40.176996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:50.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:50.576873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:14:50.587 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:14:50.585859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:00.911 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:00.909080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:00.924 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:00.922737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:11.147 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:11.144934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:11.159 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:11.158233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:21.629 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:21.627344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:21.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:21.637649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:31.871 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:31.868257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:31.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:31.881527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:42.097 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:42.095299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:42.108 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:42.106616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:52.401 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:52.398911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:15:52.414 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:15:52.411020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:02.628 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:02.626196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:02.641 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:02.639799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:12.858 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:12.856155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:12.869 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:12.867676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:23.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:23.188313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:23.201 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:23.199443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:33.419 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:33.417333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:33.431 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:33.429668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:43.650 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:43.648316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:43.658 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:43.656864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:53.933 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:53.931748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:16:53.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:16:53.940194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:04.158 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:04.156065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:04.171 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:04.169457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:14.393 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:14.391313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:14.406 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:14.404216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:24.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:24.625196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:24.640 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:24.638545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:34.878 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:34.876045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:34.890 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:34.888656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:45.107 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:45.105275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:45.161 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:45.159729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:55.490 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:55.488027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:17:55.501 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:17:55.499806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:05.722 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:05.719635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:05.734 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:05.732526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:15.956 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:15.953860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:15.969 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:15.967279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:26.188 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:26.185678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:26.200 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:26.198429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:36.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:36.420459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:36.434 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:36.432345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:46.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:46.576278 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 02:18:46.569839274 +0000 UTC m=+3254.708378207 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:18:46.651 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:46.649004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:46.661 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:46.659420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:56.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:56.877478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:18:56.890 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:18:56.888251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:07.129 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:07.127199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:07.138 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:07.136298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:17.359 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:17.356744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:17.371 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:17.369228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:27.617 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:27.614871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:27.640 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:27.637737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:37.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:37.880957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:37.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:37.894152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:48.112 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:48.109984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:48.124 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:48.122464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:58.420 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:58.417635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:19:58.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:19:58.431061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:08.652 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:08.649525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:08.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:08.662558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:18.887 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:18.884680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:18.899 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:18.897616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:29.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:29.153037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:29.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:29.165170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:39.391 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:39.388945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:39.404 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:39.401714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:49.624 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:49.621567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:49.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:49.633275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:59.912 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:59.910035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:20:59.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:20:59.920518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:10.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:10.141793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:10.158 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:10.155582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:20.380 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:20.377170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:20.393 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:20.390954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:30.630 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:30.627489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:30.641 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:30.638630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:40.918 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:40.915675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:40.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:40.947479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:51.177 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:51.175066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:21:51.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:21:51.188306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:01.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:01.500505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:01.515 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:01.511618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:11.752 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:11.750067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:11.766 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:11.764025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:21.989 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:21.986084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:22.006 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:21.998774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:32.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:32.444167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:32.457 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:32.454340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:42.692 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:42.689676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:42.703 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:42.700564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:52.926 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:52.922931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:22:52.937 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:22:52.934067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:03.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:03.276612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:03.297 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:03.294728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:06.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:06.430261 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T02:23:07.062 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:07.058172 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:23:07.034+0000 7f2557ec1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:23:07.063 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:23:07.034+0000 7f2557ec1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:23:07.063 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:23:07.034+0000 7f2557ec1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:23:07.064 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:23:07.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:07.058204 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:23:07.034+0000 7f2557ec1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:23:07.065 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:23:07.034+0000 7f2557ec1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:23:07.065 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:23:07.034+0000 7f2557ec1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:23:07.066 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:23:13.536 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:13.533187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:13.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:13.550683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:23.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:23.789487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:23.801 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:23.799290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:34.083 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:34.080785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:34.177 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:34.173291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:44.431 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:44.428498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:44.443 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:44.440570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:54.702 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:54.699020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:23:54.716 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:23:54.713332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:04.938 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:04.935709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:04.951 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:04.948052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:15.168 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:15.165981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:15.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:15.177473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:25.403 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:25.400208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:25.413 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:25.410006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:35.633 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:35.631014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:35.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:35.644754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:45.873 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:45.870677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:45.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:45.881782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:56.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:56.102913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:24:56.116 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:24:56.114224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:06.335 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:06.333360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:06.347 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:06.345912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:16.601 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:16.595198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:16.606 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:16.603758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:26.825 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:26.824267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:26.838 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:26.836773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:37.052 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:37.051603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:37.067 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:37.067082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:47.286 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:47.285631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:47.303 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:47.299750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:57.518 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:57.517881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:25:57.531 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:25:57.531087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:07.754 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:07.753597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:07.766 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:07.765295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:17.986 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:17.985873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:17.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:17.998953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:28.221 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:28.221091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:28.231 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:28.231479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:38.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:38.453776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:38.465 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:38.466251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:48.686 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:48.686631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:48.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:48.698886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:58.919 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:58.920177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:26:58.980 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:26:58.981040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:09.199 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:09.200006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:09.212 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:09.213420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:19.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:19.434980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:19.446 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:19.447735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:29.710 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:29.712103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:29.738 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:29.735151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:39.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:39.959935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:39.970 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:39.972572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:50.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:50.191572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:27:50.201 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:27:50.203834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:00.423 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:00.425218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:00.436 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:00.438306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:10.654 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:10.656729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:10.665 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:10.667771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:20.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:20.886008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:20.912 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:20.914354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:31.148 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:31.150606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:31.160 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:31.162733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:41.382 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:41.385018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:41.395 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:41.398473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:51.619 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:51.621915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:28:51.631 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:28:51.634321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:01.895 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:01.898600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:01.907 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:01.910595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:12.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:12.146863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:12.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:12.160331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:22.426 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:22.429805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:22.437 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:22.441464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:32.657 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:32.660139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:32.670 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:32.673500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:42.889 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:42.892847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:42.900 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:42.902993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:53.130 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:53.132883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:29:53.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:29:53.143163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:03.354 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:03.358338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:03.367 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:03.371167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:13.589 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:13.593419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:13.602 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:13.606760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:23.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:23.826208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:23.835 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:23.839912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:34.052 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:34.056776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:34.072 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:34.068634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:44.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:44.284857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:44.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:44.295822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:54.536 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:54.540263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:30:54.544 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:30:54.549209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:04.767 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:04.772043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:04.778 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:04.782554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:14.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:14.998442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:15.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:15.008821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:25.248 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:25.252982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:25.297 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:25.301105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:35.555 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:35.560195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:35.569 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:35.574015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:45.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:45.796065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:45.803 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:45.808413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:56.023 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:56.029071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:31:56.035 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:31:56.040845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:06.308 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:06.313381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:06.321 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:06.326412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:16.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:16.557908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:16.568 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:16.572966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:26.785 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:26.790661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:26.799 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:26.803834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:37.022 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:37.027888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:37.035 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:37.040712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:47.255 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:47.261308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:47.268 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:47.274186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:57.492 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:57.496980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:32:57.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:32:57.509132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:07.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:07.726962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:07.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:07.738953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:17.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:17.960486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:17.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:17.974257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:28.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:28.195024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:28.202 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:28.208418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:38.423 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:38.429963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:38.439 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:38.445608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:48.660 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:48.666791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:48.677 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:48.682343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:58.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:58.902420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:33:58.908 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:33:58.914997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:09.171 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:09.178130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:09.186 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:09.191913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:19.404 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:19.411299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:19.420 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:19.425450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:29.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:29.646316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:29.650 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:29.655114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:39.900 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:39.907051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:39.913 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:39.920667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:50.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:50.145771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:34:50.147 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:34:50.154447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:00.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:00.462680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:00.457 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:00.463458 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 02:35:00.450230667 +0000 UTC m=+4228.588769590 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:35:00.462 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:00.469917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:10.682 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:10.687231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:10.696 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:10.703351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:20.934 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:20.941237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:20.946 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:20.953440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:31.167 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:31.174854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:31.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:31.186661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:41.400 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:41.407831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:41.412 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:41.420267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:51.630 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:51.637148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:35:51.645 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:35:51.652744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:01.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:01.970848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:01.978 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:01.985938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:12.251 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:12.259225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:12.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:12.271365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:22.482 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:22.490103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:22.494 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:22.501830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:32.712 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:32.720123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:32.726 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:32.734028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:42.941 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:42.948828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:42.951 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:42.959873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:53.173 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:53.181394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:36:53.186 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:36:53.193963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:03.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:03.409452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:03.412 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:03.421004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:13.651 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:13.659800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:13.660 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:13.668831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:23.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:23.888060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:23.887 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:23.896113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:34.107 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:34.115195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:34.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:34.128784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:44.342 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:44.350069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:44.354 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:44.362938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:54.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:54.612989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:37:54.614 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:37:54.622856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:04.832 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:04.840313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:04.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:04.853997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:15.072 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:15.081054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:15.086 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:15.095213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:25.309 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:25.317148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:25.324 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:25.332305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:35.545 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:35.553686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:35.559 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:35.567918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:45.777 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:45.786086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:45.824 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:45.832676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:56.045 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:56.053873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:38:56.059 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:38:56.067846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:06.340 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:06.348369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:06.386 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:06.394965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:16.605 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:16.614563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:16.615 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:16.624686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:26.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:26.843143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:26.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:26.858624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:37.100 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:37.109594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:37.113 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:37.122684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:47.057 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:47.066238 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T02:39:47.330 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:47.339385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:47.342 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:47.351707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:47.682 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:47.690845 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:39:47.684+0000 7f4470fdf340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:39:47.682 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:39:47.684+0000 7f4470fdf340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:39:47.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:39:47.684+0000 7f4470fdf340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:39:47.683 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:39:47.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:47.690872 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:39:47.684+0000 7f4470fdf340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:39:47.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:39:47.684+0000 7f4470fdf340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:39:47.684 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:39:47.684+0000 7f4470fdf340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:39:47.684 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:39:57.566 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:57.574974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:39:57.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:39:57.585876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:07.798 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:07.807295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:07.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:07.822279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:18.036 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:18.045147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:18.049 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:18.058976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:28.270 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:28.278688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:28.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:28.292275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:38.544 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:38.553639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:38.556 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:38.565687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:48.779 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:48.788506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:48.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:48.802192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:59.014 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:59.023670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:40:59.025 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:40:59.035272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:09.247 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:09.257136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:09.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:09.267008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:19.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:19.530214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:19.532 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:19.541714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:29.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:29.765702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:29.771 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:29.779419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:39.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:40.007910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:40.012 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:40.022280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:50.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:50.245059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:41:50.249 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:41:50.259174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:00.473 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:00.483093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:00.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:00.495488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:10.706 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:10.715950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:10.715 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:10.725862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:20.938 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:20.947852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:20.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:20.960841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:31.169 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:31.179044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:31.181 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:31.192035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:41.403 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:41.413930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:41.417 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:41.427533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:51.678 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:51.688451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:42:51.694 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:42:51.704664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:01.959 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:01.969909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:01.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:01.979103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:12.185 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:12.196070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:12.207 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:12.218193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:22.495 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:22.506012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:22.508 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:22.518652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:32.744 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:32.754430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:32.826 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:32.782368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:43.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:43.015002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:43.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:43.027245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:53.288 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:53.298873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:43:53.311 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:43:53.322049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:03.532 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:03.543199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:03.566 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:03.577821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:13.789 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:13.800256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:13.802 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:13.813225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:24.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:24.030974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:24.030 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:24.041313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:34.271 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:34.280474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:34.279 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:34.290302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:44.501 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:44.511492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:44.515 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:44.526465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:54.735 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:54.746610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:44:54.746 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:44:54.757701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:04.988 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:04.999688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:04.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:05.009452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:15.243 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:15.254341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:15.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:15.268323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:25.476 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:25.487633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:25.487 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:25.498592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:35.715 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:35.726463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:35.725 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:35.736405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:45.949 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:45.960315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:45.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:45.973865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:56.178 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:56.188411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:45:56.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:45:56.201485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:06.411 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:06.421889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:06.422 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:06.433967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:16.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:16.651165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:16.648 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:16.659947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:26.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:26.878430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:26.881 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:26.892995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:37.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:37.115108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:37.115 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:37.127231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:47.364 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:47.376103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:47.391 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:47.401337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:57.663 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:57.674364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:46:57.672 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:46:57.684495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:07.893 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:07.904451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:07.906 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:07.918062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:18.126 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:18.137221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:18.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:18.202446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:28.409 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:28.421045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:28.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:28.432966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:38.640 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:38.651605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:38.651 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:38.663475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:48.872 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:48.884255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:48.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:48.896935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:59.126 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:59.137972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:47:59.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:47:59.151270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:09.386 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:09.397875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:09.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:09.413209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:19.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:19.632995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:19.632 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:19.644227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:29.852 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:29.863873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:29.863 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:29.875552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:40.085 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:40.096716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:40.099 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:40.111361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:50.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:50.334000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:48:50.335 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:48:50.347791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:00.561 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:00.573538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:00.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:00.587243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:10.806 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:10.818444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:10.815 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:10.827732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:21.049 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:21.061696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:21.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:21.078192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:31.286 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:31.297932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:31.298 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:31.311128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:41.521 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:41.533689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:41.534 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:41.546990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:51.756 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:51.768640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:49:51.767 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:49:51.779673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:01.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:02.004759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:02.003 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:02.016238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:12.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:12.236503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:12.231 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:12.244402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:22.450 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:22.463072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:22.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:22.473757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:32.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:32.696277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:32.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:32.710963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:42.916 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:42.928233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:42.930 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:42.942190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:53.148 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:53.160454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:50:53.160 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:50:53.172181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:03.374 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:03.387215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:03.388 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:03.401026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:13.608 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:13.620705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:13.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:13.633949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:23.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:23.858555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:23.853 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:23.866980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:34.077 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:34.090078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:34.090 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:34.102867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:44.309 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:44.322243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:44.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:44.334585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:54.540 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:54.553666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:51:54.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:51:54.566659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:04.774 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:04.787449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:04.787 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:04.800327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:15.006 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:15.018862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:15.019 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:15.031175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:15.237 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:15.248720 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 02:52:15.191011209 +0000 UTC m=+5263.329550207 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:52:25.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:25.257594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:25.253 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:25.266159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:35.475 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:35.488419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:35.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:35.499350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:45.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:45.720122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:45.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:45.732676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:55.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:55.949413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:52:55.949 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:52:55.962220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:06.200 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:06.213433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:06.211 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:06.224778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:16.557 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:16.570588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:16.568 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:16.581391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:26.788 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:26.801226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:26.801 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:26.814818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:37.021 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:37.034314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:37.033 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:37.047347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:47.254 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:47.267649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:47.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:47.280325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:57.487 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:57.500117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:53:57.499 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:53:57.513295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:07.723 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:07.737077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:07.735 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:07.749124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:17.956 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:17.969956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:17.969 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:17.983527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:28.194 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:28.206848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:28.205 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:28.219267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:38.423 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:38.436864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:38.462 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:38.475592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:48.685 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:48.698783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:48.702 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:48.716154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:58.920 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:58.934100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:54:58.933 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:54:58.947516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:09.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:09.204049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:09.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:09.218207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:19.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:19.439430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:19.439 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:19.452752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:29.656 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:29.670099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:29.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:29.678295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:39.902 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:39.915508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:39.910 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:39.923910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:50.133 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:50.147543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:55:50.145 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:55:50.159067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:00.364 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:00.377716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:00.377 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:00.389285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:10.622 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:10.636389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:10.633 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:10.646752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:20.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:20.859428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:20.857 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:20.871280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:27.685 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:27.699677 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T02:56:28.355 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:28.368448 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:56:28.358+0000 7ff6d68fb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:56:28.355 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:56:28.358+0000 7ff6d68fb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:56:28.356 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:56:28.358+0000 7ff6d68fb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:56:28.356 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:56:28.356 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:28.368475 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:56:28.358+0000 7ff6d68fb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:56:28.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:56:28.358+0000 7ff6d68fb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:56:28.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T02:56:28.358+0000 7ff6d68fb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:56:28.357 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:56:31.094 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:31.108172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:31.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:31.118422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:41.325 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:41.339614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:41.334 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:41.349057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:51.566 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:51.580686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:56:51.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:56:51.590170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:01.797 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:01.809901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:01.805 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:01.819768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:12.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:12.060694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:12.060 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:12.073468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:22.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:22.336296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:22.333 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:22.347506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:32.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:32.567548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:32.567 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:32.581703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:42.786 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:42.800068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:42.800 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:42.813614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:53.017 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:53.031217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:57:53.029 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:57:53.043431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:03.254 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:03.268363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:03.266 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:03.280471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:13.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:13.499353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:13.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:13.511453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:23.720 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:23.734297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:23.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:23.746681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:33.948 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:33.962036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:33.959 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:33.973407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:44.173 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:44.186735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:44.185 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:44.199696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:54.404 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:54.417753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:58:54.415 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:58:54.427198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:04.632 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:04.645144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:04.645 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:04.658433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:14.894 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:14.906711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:14.907 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:14.920815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:25.160 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:25.172983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:25.170 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:25.182917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:35.386 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:35.399144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:35.396 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:35.408973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:45.615 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:45.627945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:45.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:45.640595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:55.876 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:55.888254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T02:59:55.887 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 02:59:55.899815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:06.110 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:06.121813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:06.122 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:06.134145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:16.346 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:16.357455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:16.358 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:16.370314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:26.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:26.593438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:26.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:26.607568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:36.816 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:36.828288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:36.829 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:36.841518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:47.047 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:47.058450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:47.057 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:47.069368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:57.277 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:57.286480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:00:57.291 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:00:57.302834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:07.532 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:07.542891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:07.541 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:07.552734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:17.758 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:17.769144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:17.771 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:17.782543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:27.991 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:28.002290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:28.005 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:28.016125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:28.005 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:28.016164 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:38.236 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:38.246841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:38.250 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:38.261349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:48.471 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:48.481564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:48.482 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:48.492596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:58.701 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:58.711944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:01:58.714 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:01:58.723730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:08.933 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:08.943675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:08.948 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:08.956612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:19.166 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:19.175843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:19.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:19.185086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:29.426 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:29.433910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:29.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:29.449730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:39.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:39.669555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:39.670 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:39.679428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:49.888 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:49.898148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:02:49.904 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:02:49.913784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:00.148 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:00.157752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:00.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:00.166960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:10.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:10.416668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:10.416 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:10.425758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:20.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:20.691621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:20.694 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:20.703396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:30.988 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:30.997582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:31.041 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:31.050469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:41.296 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:41.305789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:41.309 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:41.318527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:51.536 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:51.545097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:03:51.550 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:03:51.558962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:01.771 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:01.780326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:01.782 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:01.791379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:12.002 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:12.011094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:12.016 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:12.025496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:22.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:22.246381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:22.248 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:22.257649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:32.468 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:32.476750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:32.477 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:32.486096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:42.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:42.707248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:42.711 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:42.720329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:52.934 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:52.942543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:04:52.948 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:04:52.956976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:03.170 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:03.178814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:03.180 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:03.189101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:13.401 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:13.408920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:13.412 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:13.420395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:23.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:23.645365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:23.652 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:23.660469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:33.973 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:33.981069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:33.984 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:33.991286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:44.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:44.244319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:44.245 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:44.253748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:54.468 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:54.475877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:05:54.480 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:05:54.486698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:04.911 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:04.918777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:04.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:04.930682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:15.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:15.150314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:15.154 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:15.161772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:25.374 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:25.381184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:25.387 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:25.394539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:35.638 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:35.645261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:35.651 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:35.658302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:45.871 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:45.878547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:45.882 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:45.890136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:56.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:56.108196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:06:56.151 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:06:56.158395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:06.448 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:06.455428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:06.460 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:06.467038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:16.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:16.714101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:16.715 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:16.722882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:26.933 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:26.940371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:26.946 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:26.953123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:37.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:37.644270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:37.649 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:37.656241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:47.868 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:47.875222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:47.913 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:47.920413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:58.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:58.144509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:07:58.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:07:58.211628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:08.701 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:08.707655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:08.709 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:08.716526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:18.984 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:18.991346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:19.009 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:19.015484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:29.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:29.185434 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 03:08:29.177442537 +0000 UTC m=+6237.315981512 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:08:29.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:29.241282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:29.317 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:29.323042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:39.561 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:39.568049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:39.571 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:39.577819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:49.793 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:49.799423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:08:49.806 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:08:49.812856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:00.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:00.034175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:00.040 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:00.046828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:10.292 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:10.298095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:10.303 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:10.309734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:20.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:20.526567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:20.533 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:20.539217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:30.748 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:30.753842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:30.758 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:30.765089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:40.983 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:40.983517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:40.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:41.000418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:51.244 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:51.249923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:09:51.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:09:51.265503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:01.482 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:01.487701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:01.496 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:01.502110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:11.715 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:11.721079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:11.729 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:11.734981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:21.948 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:21.953880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:21.960 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:21.966495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:32.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:32.181229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:32.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:32.193999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:42.406 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:42.412288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:42.418 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:42.424544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:52.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:52.642803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:10:52.649 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:10:52.655538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:02.871 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:02.876458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:02.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:02.890442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:13.106 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:13.111142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:13.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:13.126507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:23.344 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:23.349673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:23.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:23.362433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:33.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:33.601601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:33.608 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:33.614025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:43.833 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:43.838808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:43.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:43.851376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:54.067 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:54.072367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:11:54.081 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:11:54.086741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:04.328 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:04.333175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:04.336 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:04.341724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:14.555 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:14.560213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:14.568 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:14.573661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:24.830 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:24.835884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:24.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:24.849387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:35.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:35.069231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:35.074 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:35.080041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:45.290 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:45.294953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:45.301 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:45.306558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:55.524 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:55.529269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:12:55.538 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:12:55.542740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:05.761 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:05.765822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:05.774 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:05.779574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:08.371 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:08.376252 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T03:13:09.019 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:09.023745 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:13:09.013+0000 7fb17c788340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:13:09.019 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:13:09.013+0000 7fb17c788340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:13:09.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:13:09.013+0000 7fb17c788340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:13:09.020 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:13:09.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:09.023779 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:13:09.013+0000 7fb17c788340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:13:09.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:13:09.013+0000 7fb17c788340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:13:09.021 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:13:09.013+0000 7fb17c788340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:13:09.021 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:13:15.995 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:16.000630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:16.007 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:16.012293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:26.230 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:26.233549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:26.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:26.246022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:36.464 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:36.468516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:36.475 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:36.479760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:46.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:46.698208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:46.701 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:46.707077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:56.918 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:56.922666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:13:56.930 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:13:56.935598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:07.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:07.183840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:07.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:07.196453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:17.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:17.411934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:17.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:17.425566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:27.655 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:27.659642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:27.666 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:27.670332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:37.886 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:37.891028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:37.947 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:37.951578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:48.165 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:48.169349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:48.176 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:48.180927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:58.450 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:58.455326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:14:58.462 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:14:58.467226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:08.717 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:08.721745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:08.728 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:08.733728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:18.944 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:18.948695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:18.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:18.962646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:29.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:29.183555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:29.192 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:29.197336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:39.419 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:39.422794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:39.431 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:39.436213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:49.651 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:49.656019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:49.661 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:49.665960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:59.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:59.883996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:15:59.891 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:15:59.895468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:10.107 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:10.111155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:10.119 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:10.124069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:20.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:20.361756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:20.368 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:20.373174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:30.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:30.594685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:30.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:30.608822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:40.858 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:40.862189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:40.869 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:40.874073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:51.095 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:51.099754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:16:51.107 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:16:51.111163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:01.326 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:01.330518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:01.339 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:01.344026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:11.559 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:11.563529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:11.570 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:11.575259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:21.796 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:21.800742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:21.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:21.812070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:32.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:32.032403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:32.040 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:32.044663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:42.260 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:42.264379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:42.271 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:42.276264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:52.491 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:52.494448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:17:52.501 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:17:52.505809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:02.749 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:02.753544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:02.759 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:02.763465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:12.997 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:13.001857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:13.039 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:13.043389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:23.288 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:23.292294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:23.300 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:23.305062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:33.524 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:33.528238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:33.547 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:33.551757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:43.793 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:43.797483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:43.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:43.811929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:54.026 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:54.030651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:18:54.035 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:18:54.039706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:04.253 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:04.256963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:04.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:04.271856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:14.493 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:14.497258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:14.506 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:14.510520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:24.726 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:24.730520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:24.742 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:24.746084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:34.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:34.965716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:34.973 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:34.977389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:45.197 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:45.200673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:45.209 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:45.213392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:55.450 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:55.453823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:19:55.464 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:19:55.468185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:05.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:05.735255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:05.750 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:05.748398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:15.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:15.966784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:15.980 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:15.983775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:26.199 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:26.202336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:26.211 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:26.215187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:36.432 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:36.435992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:36.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:36.444319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:46.668 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:46.672247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:46.676 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:46.680471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:56.932 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:56.936421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:20:56.943 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:20:56.947179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:07.159 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:07.162588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:07.171 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:07.175259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:17.394 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:17.398304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:17.406 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:17.410153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:27.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:27.630893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:27.640 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:27.644012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:37.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:37.889003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:37.900 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:37.904246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:48.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:48.124597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:48.134 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:48.136825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:58.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:58.360749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:21:58.370 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:21:58.373339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:08.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:08.591251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:08.600 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:08.603789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:18.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:18.840938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:18.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:18.853512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:29.067 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:29.070831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:29.077 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:29.080987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:39.318 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:39.321618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:39.360 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:39.364128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:49.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:49.585148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:49.594 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:49.598019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:59.819 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:59.822197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:22:59.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:22:59.830872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:10.041 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:10.045185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:10.050 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:10.053978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:20.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:20.270737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:20.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:20.285200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:30.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:30.507295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:30.516 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:30.520336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:40.739 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:40.742122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:40.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:40.756722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:50.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:50.976115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:23:50.985 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:23:50.989046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:01.203 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:01.206500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:01.268 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:01.272605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:11.486 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:11.489169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:11.505 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:11.504620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:21.720 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:21.724212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:21.734 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:21.737171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:31.953 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:31.956460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:31.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:31.966845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:42.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:42.186963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:42.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:42.197750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:52.423 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:52.426748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:24:52.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:24:52.437192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:02.658 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:02.661789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:02.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:02.673029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:12.887 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:12.891238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:12.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:12.901541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:23.119 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:23.120319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:23.130 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:23.134455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:33.344 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:33.347753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:33.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:33.361041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:43.599 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:43.602733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:43.610 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:43.613896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:43.882 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:43.885985 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 03:25:43.877837758 +0000 UTC m=+7272.016376735 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:25:53.862 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:53.865774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:25:53.875 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:25:53.878733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:04.093 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:04.096643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:04.104 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:04.108927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:14.345 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:14.348584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:14.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:14.359907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:24.574 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:24.578051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:24.602 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:24.606784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:34.823 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:34.827224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:34.835 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:34.838770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:45.056 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:45.059664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:45.071 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:45.074159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:55.339 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:55.343152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:26:55.349 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:26:55.352767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:05.568 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:05.571432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:05.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:05.580964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:15.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:15.795913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:15.804 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:15.808047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:26.018 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:26.021633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:26.029 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:26.033376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:36.247 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:36.250994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:36.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:36.261583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:46.472 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:46.475537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:46.482 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:46.485606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:56.699 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:56.703093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:27:56.711 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:27:56.714750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:06.931 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:06.933431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:06.943 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:06.946953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:17.168 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:17.172230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:17.179 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:17.183554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:27.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:27.444249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:27.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:27.457719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:37.678 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:37.677988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:37.692 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:37.696064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:47.911 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:47.914920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:47.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:47.926287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:58.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:58.146322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:28:58.154 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:28:58.158005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:08.372 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:08.375238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:08.384 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:08.387776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:18.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:18.607919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:18.619 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:18.622813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:28.842 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:28.845838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:28.859 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:28.863243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:39.081 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:39.084875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:39.094 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:39.097766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:49.030 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:49.033750 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T03:29:49.314 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:49.318197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:49.325 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:49.327774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:49.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:49.847943 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:29:49.839+0000 7f89bc4fb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:29:49.845 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:29:49.839+0000 7f89bc4fb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:29:49.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:29:49.839+0000 7f89bc4fb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:29:49.846 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:29:49.847 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:49.847969 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:29:49.839+0000 7f89bc4fb340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:29:49.847 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:29:49.839+0000 7f89bc4fb340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:29:49.847 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:29:49.839+0000 7f89bc4fb340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:29:49.848 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:29:59.546 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:59.550051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:29:59.554 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:29:59.558148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:09.786 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:09.789685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:09.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:09.798601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:20.026 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:20.030218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:20.038 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:20.041778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:30.256 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:30.259701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:30.268 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:30.271883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:40.548 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:40.551637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:40.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:40.581995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:50.799 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:50.801863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:30:50.812 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:30:50.816108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:01.029 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:01.033344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:01.038 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:01.042197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:11.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:11.262878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:11.273 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:11.276726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:21.493 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:21.497055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:21.503 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:21.507404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:31.725 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:31.728368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:31.738 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:31.742569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:41.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:41.967162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:41.973 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:41.977161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:52.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:52.227914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:31:52.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:31:52.238828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:02.460 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:02.464294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:02.469 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:02.473566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:12.687 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:12.691295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:12.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:12.702473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:22.966 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:22.969598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:22.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:22.978784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:33.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:33.218050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:33.227 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:33.230202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:43.448 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:43.451377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:43.460 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:43.463542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:53.677 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:53.681323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:32:53.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:32:53.692669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:03.910 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:03.913839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:03.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:03.925540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:14.149 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:14.151886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:14.160 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:14.163915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:24.385 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:24.388725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:24.397 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:24.400367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:34.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:34.624544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:34.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:34.637822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:44.857 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:44.860169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:44.868 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:44.871737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:55.091 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:55.094598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:33:55.103 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:33:55.106084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:05.325 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:05.327867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:05.338 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:05.341564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:15.568 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:15.571328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:15.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:15.582972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:25.856 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:25.859210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:25.902 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:25.904739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:36.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:36.181408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:36.199 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:36.193500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:46.447 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:46.450198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:46.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:46.463833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:56.682 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:56.680537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:34:56.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:34:56.693734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:06.935 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:06.937869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:06.947 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:06.949916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:17.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:17.216682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:17.226 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:17.228798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:27.443 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:27.445548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:27.454 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:27.457330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:37.674 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:37.676358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:37.688 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:37.690848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:47.907 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:47.909694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:47.920 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:47.922152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:58.140 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:58.142123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:35:58.176 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:35:58.178428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:08.410 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:08.412355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:08.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:08.427425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:18.645 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:18.647737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:18.657 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:18.659676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:28.878 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:28.880149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:28.890 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:28.892604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:39.108 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:39.110188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:39.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:39.122680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:49.335 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:49.336709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:49.346 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:49.348651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:59.566 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:59.567697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:36:59.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:36:59.577618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:09.794 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:09.795992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:09.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:09.808988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:20.029 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:20.030822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:20.040 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:20.041906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:30.258 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:30.259411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:30.269 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:30.271493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:40.484 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:40.486158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:40.498 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:40.499069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:50.718 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:50.719422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:37:50.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:37:50.732553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:00.948 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:00.950113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:00.959 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:00.961578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:11.177 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:11.178535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:11.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:11.188110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:21.409 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:21.409980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:21.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:21.422694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:31.656 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:31.657793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:31.666 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:31.668060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:41.886 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:41.887908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:41.898 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:41.899477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:52.142 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:52.144032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:38:52.153 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:38:52.154892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:02.374 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:02.375986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:02.385 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:02.386751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:12.603 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:12.604184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:12.615 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:12.616666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:22.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:22.834803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:22.846 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:22.847655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:33.066 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:33.067483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:33.076 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:33.078007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:43.321 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:43.322791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:43.335 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:43.336770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:53.574 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:53.575714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:39:53.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:39:53.583462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:03.821 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:03.822481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:03.833 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:03.834495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:14.058 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:14.059579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:14.070 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:14.071405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:24.288 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:24.288916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:24.300 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:24.299876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:34.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:34.520937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:34.531 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:34.532260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:44.753 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:44.754475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:44.766 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:44.767445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:54.988 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:54.988476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:40:55.000 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:40:55.000855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:05.221 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:05.221738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:05.233 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:05.234025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:15.483 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:15.483793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:15.493 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:15.493956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:25.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:25.732629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:25.744 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:25.745184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:36.013 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:36.014244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:36.025 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:36.026488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:46.248 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:46.248944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:46.263 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:46.261989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:56.482 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:56.483229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:41:56.494 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:41:56.495551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:06.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:06.721912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:06.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:06.732450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:16.948 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:16.948997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:16.959 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:16.960225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:27.181 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:27.181834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:27.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:27.190374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:37.413 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:37.413737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:37.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:37.426411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:47.643 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:47.644141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:47.715 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:47.715967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:58.011 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:58.012153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:42:58.032 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:42:58.027099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:08.243 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:08.243997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:08.252 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:08.252617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:18.472 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:18.473040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:18.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:18.485220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:28.705 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:28.706102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:28.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:28.719968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:38.946 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:38.946270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:38.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:38.959221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:49.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:49.337737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:49.349 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:49.350028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:59.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:59.565843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:59.577 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:59.576725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:43:59.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:43:59.698287 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 03:43:59.691634965 +0000 UTC m=+8367.830173950 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T03:44:09.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:09.795654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:44:09.804 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:09.804777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:44:20.223 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:20.223054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:44:20.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:20.233401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:44:30.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:30.453732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:44:30.465 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:30.465709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:44:40.685 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:40.685520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:44:40.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:40.698171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:44:50.922 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:50.922340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:44:50.935 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:44:50.935757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:01.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:01.155166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:01.170 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:01.171027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:11.393 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:11.393510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:11.404 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:11.405026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:21.729 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:21.729583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:21.739 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:21.739231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:32.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:32.138858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:32.153 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:32.153426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:42.376 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:42.375738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:42.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:42.440153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:52.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:52.659132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:45:52.671 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:45:52.671313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:02.891 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:02.891379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:02.901 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:02.901774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:13.120 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:13.119986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:13.134 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:13.134347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:23.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:23.455533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:23.466 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:23.466800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:29.855 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:29.853372 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T03:46:30.470 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:30.469693 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:46:30.458+0000 7fcb93363340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:46:30.470 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:46:30.458+0000 7fcb93363340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:46:30.471 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:46:30.458+0000 7fcb93363340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:46:30.471 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:46:30.471 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:30.469721 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:46:30.458+0000 7fcb93363340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:46:30.471 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:46:30.458+0000 7fcb93363340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:46:30.472 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T03:46:30.458+0000 7fcb93363340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:46:30.472 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:46:33.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:33.721546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:33.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:33.733976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:43.955 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:43.955337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:43.967 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:43.966247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:54.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:54.188777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:46:54.202 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:46:54.202854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:04.427 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:04.426857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:04.439 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:04.439050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:14.667 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:14.667214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:14.678 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:14.678397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:24.930 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:24.930678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:24.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:24.942145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:35.161 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:35.161322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:35.173 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:35.173399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:45.390 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:45.389687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:45.401 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:45.401366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:55.620 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:55.620238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:47:55.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:47:55.634831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:05.855 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:05.854750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:05.868 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:05.867704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:16.091 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:16.090621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:16.102 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:16.102209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:26.324 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:26.323439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:26.335 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:26.334925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:36.559 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:36.558600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:36.571 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:36.570598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:46.798 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:46.791950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:46.804 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:46.803199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:57.033 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:57.032115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:48:57.049 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:48:57.047923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:07.278 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:07.277590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:07.290 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:07.290358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:17.509 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:17.509174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:17.523 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:17.519410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:27.745 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:27.744566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:27.757 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:27.756770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:38.011 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:38.010707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:38.021 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:38.020476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:48.243 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:48.242699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:48.253 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:48.252924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:58.491 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:58.491179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:49:58.507 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:49:58.507754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:08.726 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:08.725663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:08.769 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:08.767741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:18.986 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:18.985784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:18.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:18.997714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:29.253 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:29.252985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:29.264 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:29.264308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:39.484 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:39.483874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:39.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:39.495731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:49.712 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:49.711757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:49.725 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:49.724016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:59.946 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:59.945835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:50:59.956 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:50:59.955734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:10.182 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:10.181738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:10.207 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:10.206574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:20.428 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:20.427051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:20.441 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:20.440699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:30.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:30.664141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:30.672 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:30.672232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:40.900 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:40.899450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:40.912 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:40.911674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:51.192 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:51.191372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:51:51.206 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:51:51.205338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:01.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:01.422858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:01.435 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:01.433948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:11.685 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:11.684553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:11.694 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:11.692798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:21.926 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:21.925988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:21.935 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:21.934417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:32.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:32.192345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:32.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:32.203729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:42.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:42.420002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:42.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:42.432708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:52.659 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:52.653294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:52:52.667 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:52:52.666408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:02.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:02.882494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:02.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:02.895568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:13.118 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:13.117165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:13.131 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:13.129174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:23.351 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:23.350724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:23.364 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:23.363071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:33.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:33.581608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:33.595 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:33.594778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:43.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:43.812407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:43.828 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:43.827557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:54.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:54.045145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:53:54.060 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:53:54.059179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:04.297 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:04.296558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:04.312 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:04.311835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:14.563 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:14.562188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:14.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:14.574361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:24.815 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:24.814751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:24.830 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:24.829222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:35.043 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:35.042655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:35.057 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:35.055256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:45.275 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:45.274276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:45.288 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:45.287626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:55.508 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:55.506849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:54:55.520 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:54:55.519028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:05.741 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:05.740354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:05.753 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:05.752179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:15.973 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:15.972042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:15.984 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:15.983419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:26.226 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:26.225500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:26.238 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:26.238164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:36.459 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:36.458036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:36.470 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:36.468847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:46.686 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:46.685743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:46.697 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:46.695569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:56.916 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:56.914946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:55:56.929 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:55:56.929196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:07.145 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:07.144639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:07.158 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:07.156869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:17.390 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:17.387769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:17.827 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:17.397666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:27.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:27.634180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:27.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:27.646377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:37.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:37.866253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:37.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:37.878016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:48.105 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:48.104173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:48.117 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:48.116627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:58.347 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:58.346399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:56:58.356 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:56:58.356072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:08.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:08.574431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:08.589 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:08.588068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:18.850 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:18.848888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:18.864 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:18.860512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:29.084 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:29.083109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:29.095 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:29.095035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:39.315 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:39.313994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:39.327 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:39.326039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:49.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:49.564019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:49.593 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:49.592322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:59.817 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:59.815710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:57:59.830 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:57:59.829786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:10.060 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:10.059537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:10.073 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:10.071989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:20.292 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:20.291273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:20.304 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:20.303429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:30.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:30.523609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:30.536 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:30.535216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:40.799 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:40.798249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:40.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:40.811761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:51.106 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:51.104539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:58:51.118 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:58:51.117606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:01.338 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:01.337241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:01.348 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:01.347649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:11.571 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:11.570010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:11.585 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:11.584488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:21.812 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:21.809028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:21.821 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:21.820280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:32.040 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:32.039364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:32.052 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:32.051444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:42.272 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:42.270630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:42.283 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:42.282115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:52.502 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:52.501115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T03:59:52.514 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 03:59:52.513035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:02.733 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:02.732364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:02.759 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:02.758007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:12.979 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:12.977783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:13.080 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:13.079546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:23.372 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:23.370675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:23.387 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:23.386535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:33.608 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:33.606988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:33.642 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:33.630239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:43.853 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:43.852280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:43.864 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:43.863893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:54.100 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:54.098761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:00:54.114 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:00:54.113056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:04.333 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:04.330981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:04.343 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:04.342856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:14.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:14.646107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:14.658 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:14.657430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:15.078 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:15.077729 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 04:01:15.058143355 +0000 UTC m=+9403.196682342 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T04:01:24.874 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:24.872622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:24.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:24.882167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:35.149 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:35.148504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:35.158 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:35.157034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:35.159 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:35.157060 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:45.695 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:45.694240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:45.706 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:45.705696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:55.934 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:55.932636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:01:55.946 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:01:55.945264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:06.165 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:06.164159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:06.177 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:06.175879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:16.502 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:16.501112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:16.806 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:16.805030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:27.032 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:27.030972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:27.042 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:27.041179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:37.262 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:37.260561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:37.272 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:37.271039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:47.494 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:47.492540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:47.505 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:47.503794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:57.727 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:57.725178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:02:57.739 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:02:57.737834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:07.997 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:07.995769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:08.010 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:08.009284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:10.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:10.476597 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T04:03:11.074 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:11.072506 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:03:11.064+0000 7f692d2db340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:03:11.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:03:11.064+0000 7f692d2db340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:03:11.075 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:03:11.064+0000 7f692d2db340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:03:11.075 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:03:11.076 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:11.072533 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:03:11.064+0000 7f692d2db340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:03:11.076 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:03:11.064+0000 7f692d2db340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:03:11.076 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:03:11.064+0000 7f692d2db340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:03:11.076 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:03:18.231 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:18.229679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:18.243 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:18.242623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:28.465 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:28.463466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:28.476 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:28.475028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:38.695 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:38.693799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:38.706 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:38.705191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:48.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:48.953274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:48.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:48.967036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:59.190 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:59.188199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:03:59.201 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:03:59.199923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:09.424 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:09.422458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:09.436 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:09.434683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:19.660 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:19.657892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:19.671 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:19.670182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:29.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:29.883708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:29.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:29.895877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:40.118 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:40.116352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:40.128 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:40.126867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:50.371 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:50.369428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:04:50.383 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:04:50.382242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:00.643 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:00.641446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:00.653 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:00.652431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:10.873 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:10.871914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:10.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:10.884281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:21.105 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:21.103888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:21.118 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:21.115984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:31.333 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:31.332004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:31.344 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:31.342353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:41.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:41.563273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:41.648 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:41.646880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:51.870 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:51.868639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:05:51.882 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:05:51.881044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:02.103 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:02.101035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:02.115 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:02.114135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:12.350 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:12.348729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:12.362 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:12.360157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:22.587 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:22.586249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:22.600 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:22.599405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:32.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:32.820225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:32.836 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:32.834921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:43.060 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:43.058790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:43.074 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:43.072441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:53.294 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:53.292156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:06:53.306 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:06:53.304414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:03.551 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:03.549789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:03.563 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:03.561382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:13.780 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:13.778664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:13.793 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:13.790264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:24.012 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:24.011261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:24.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:24.019635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:34.248 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:34.246547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:34.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:34.258184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:44.487 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:44.485831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:44.498 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:44.497874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:54.717 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:54.715814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:07:54.728 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:07:54.727351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:04.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:04.948522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:04.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:04.960936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:15.185 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:15.184145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:15.198 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:15.197022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:25.465 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:25.463402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:25.477 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:25.475910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:35.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:35.706370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:35.720 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:35.719329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:45.947 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:45.946408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:45.960 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:45.959235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:56.181 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:56.179655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:08:56.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:08:56.192191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:06.412 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:06.410540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:06.425 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:06.424426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:16.644 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:16.643062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:16.656 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:16.655138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:26.876 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:26.875237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:26.889 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:26.887988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:37.137 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:37.136327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:37.149 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:37.148717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:47.367 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:47.366436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:47.458 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:47.457680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:57.726 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:57.725051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:09:57.738 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:09:57.737299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:07.984 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:07.983223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:07.997 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:07.997001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:18.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:18.212933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:18.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:18.256438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:28.479 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:28.478372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:28.491 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:28.490818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:38.747 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:38.745783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:38.759 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:38.758154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:48.982 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:48.980396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:48.995 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:48.993780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:59.219 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:59.217109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:10:59.700 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:10:59.226756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:09.448 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:09.447240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:09.501 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:09.499887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:19.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:19.720712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:19.782 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:19.781757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:30.000 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:29.999831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:30.011 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:30.010197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:40.225 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:40.224280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:40.242 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:40.241360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:50.492 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:50.491391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:11:50.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:11:50.503974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:00.728 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:00.727539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:00.738 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:00.737469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:10.960 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:10.959675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:10.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:10.972063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:21.195 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:21.194647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:21.208 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:21.207626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:31.436 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:31.435076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:31.444 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:31.444109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:41.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:41.692505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:41.703 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:41.702574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:51.940 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:51.939332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:12:51.951 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:12:51.950496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:02.182 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:02.181913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:02.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:02.191353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:12.448 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:12.447537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:12.462 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:12.461294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:22.681 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:22.680013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:22.694 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:22.693702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:32.917 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:32.915864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:32.926 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:32.926293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:43.147 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:43.145636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:43.188 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:43.187527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:53.405 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:53.404572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:13:53.418 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:13:53.416743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:03.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:03.638779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:03.654 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:03.653516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:13.896 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:13.896253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:13.905 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:13.904969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:24.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:24.122542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:24.136 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:24.136089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:34.359 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:34.357549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:34.369 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:34.368890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:44.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:44.589516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:44.601 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:44.600966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:54.824 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:54.823090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:14:54.835 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:14:54.834822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:05.056 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:05.055642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:05.067 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:05.066742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:15.326 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:15.325524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:15.338 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:15.337994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:25.557 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:25.556351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:25.567 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:25.567201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:35.787 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:35.786396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:35.800 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:35.798725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:46.022 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:46.021518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:46.034 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:46.033255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:56.270 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:56.269530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:15:56.288 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:15:56.287940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:06.507 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:06.507340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:06.517 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:06.516868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:16.737 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:16.736399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:16.744 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:16.744512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:26.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:26.962739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:26.978 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:26.974996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:37.195 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:37.194739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:37.208 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:37.208227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:47.473 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:47.472616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:47.483 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:47.483647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:57.702 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:57.702260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:16:57.713 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:16:57.713181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:07.945 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:07.945309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:07.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:07.957812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:18.245 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:18.244986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:18.261 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:18.260594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:28.500 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:28.500377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:28.509 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:28.509498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:38.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:38.731867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:38.744 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:38.743933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:48.960 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:48.959709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:48.971 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:48.970877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:59.213 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:59.212912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:17:59.225 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:17:59.224966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:09.446 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:09.444281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:09.456 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:09.456072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:19.673 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:19.673029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:19.684 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:19.684235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:29.901 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:29.899379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:29.910 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:29.910170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:29.934 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:29.934619 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 04:18:29.906074628 +0000 UTC m=+10438.044613571 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T04:18:40.130 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:40.129433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:40.142 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:40.142561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:50.366 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:50.366311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:18:50.374 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:18:50.374786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:00.595 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:00.595196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:00.607 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:00.607052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:10.851 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:10.850394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:10.918 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:10.917626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:21.145 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:21.144789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:21.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:21.157082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:31.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:31.407135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:31.464 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:31.464020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:41.682 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:41.682354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:41.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:41.693442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:51.079 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:51.079166 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T04:19:51.679 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:51.677911 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:19:51.670+0000 7f09cd389340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:19:51.679 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:19:51.670+0000 7f09cd389340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:19:51.680 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:19:51.670+0000 7f09cd389340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:19:51.680 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:19:51.681 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:51.677934 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:19:51.670+0000 7f09cd389340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:19:51.681 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:19:51.670+0000 7f09cd389340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:19:51.681 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:19:51.670+0000 7f09cd389340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:19:51.682 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:19:51.905 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:51.905524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:19:51.917 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:19:51.917472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:02.180 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:02.180027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:02.189 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:02.189340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:12.419 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:12.419382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:12.437 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:12.437563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:22.702 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:22.702005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:22.716 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:22.715880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:32.935 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:32.934787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:32.949 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:32.948837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:43.169 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:43.169185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:43.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:43.183153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:53.403 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:53.403116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:20:53.414 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:20:53.414348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:03.637 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:03.636972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:03.656 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:03.650739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:13.872 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:13.871737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:13.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:13.885396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:24.160 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:24.159736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:24.168 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:24.168277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:34.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:34.402124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:34.501 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:34.501249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:44.738 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:44.738147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:44.748 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:44.747443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:55.038 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:55.037465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:21:55.049 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:21:55.049284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:05.316 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:05.315613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:05.325 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:05.325698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:15.548 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:15.547582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:15.560 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:15.560417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:25.850 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:25.849893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:25.860 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:25.860545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:36.160 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:36.159945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:36.169 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:36.169460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:46.385 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:46.385162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:46.399 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:46.398937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:56.654 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:56.654366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:22:56.662 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:22:56.662653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:06.880 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:06.879965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:06.893 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:06.893454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:17.116 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:17.115821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:17.130 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:17.129736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:27.474 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:27.474162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:27.490 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:27.490122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:37.799 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:37.796882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:37.807 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:37.807237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:48.032 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:48.031633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:48.043 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:48.043550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:58.266 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:58.266224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:23:58.279 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:23:58.277831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:08.547 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:08.546995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:08.558 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:08.557775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:18.780 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:18.779740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:18.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:18.792169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:29.022 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:29.014709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:29.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:29.027888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:39.247 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:39.246516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:39.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:39.258978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:49.484 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:49.484504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:49.497 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:49.495965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:59.718 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:59.718297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:24:59.730 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:24:59.729790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:09.954 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:09.949707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:09.968 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:09.959481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:20.224 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:20.224149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:20.237 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:20.237715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:30.685 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:30.684681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:30.718 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:30.718102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:41.269 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:41.268778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:41.278 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:41.278643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:51.494 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:51.494140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:25:51.506 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:25:51.506766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:01.730 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:01.730399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:01.743 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:01.743566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:12.082 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:12.082250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:12.090 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:12.090709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:22.305 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:22.304649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:22.367 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:22.366925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:32.614 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:32.613694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:32.626 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:32.626121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:42.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:42.847706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:42.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:42.863030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:53.080 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:53.080506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:26:53.114 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:26:53.094305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:03.317 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:03.317027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:03.337 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:03.337130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:13.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:13.565182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:13.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:13.576869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:23.800 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:23.798260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:23.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:23.808532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:34.029 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:34.029283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:34.038 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:34.037701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:44.258 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:44.258268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:44.271 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:44.271406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:54.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:54.621372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:27:54.632 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:27:54.632438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:04.934 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:04.933858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:04.978 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:04.978369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:15.223 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:15.223322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:15.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:15.232777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:25.480 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:25.480006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:25.491 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:25.489555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:35.708 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:35.707475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:35.720 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:35.720208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:45.940 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:45.939518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:45.951 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:45.951475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:56.180 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:56.179578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:28:56.191 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:28:56.191046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:06.431 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:06.421693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:06.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:06.438774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:16.663 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:16.663036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:16.671 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:16.671622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:26.891 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:26.891289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:26.911 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:26.910875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:37.127 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:37.126712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:37.140 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:37.140297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:47.456 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:47.455999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:47.511 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:47.510683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:57.729 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:57.729071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:29:57.743 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:29:57.743668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:08.005 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:08.004815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:08.020 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:08.020210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:18.242 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:18.242395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:18.255 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:18.255721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:28.481 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:28.481365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:28.495 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:28.494434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:38.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:38.718667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:38.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:38.732310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:48.952 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:48.951719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:48.964 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:48.964185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:59.188 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:59.187565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:30:59.199 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:30:59.199943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:09.537 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:09.536803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:09.557 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:09.556586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:19.830 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:19.830539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:19.841 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:19.841265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:30.096 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:30.095991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:30.107 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:30.107342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:40.352 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:40.351186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:40.366 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:40.363703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:50.586 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:50.585767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:31:50.597 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:31:50.596713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:00.816 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:00.815615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:00.937 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:00.935247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:11.155 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:11.154274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:11.172 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:11.167942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:21.572 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:21.571976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:21.583 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:21.582255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:31.802 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:31.802056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:31.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:31.813750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:42.051 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:42.051022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:42.059 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:42.059521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:42.128 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:42.127073 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 04:32:42.119159109 +0000 UTC m=+11290.257698040 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T04:32:52.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:52.437627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:32:52.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:32:52.449100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:02.705 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:02.703912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:02.717 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:02.717105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:12.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:12.935989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:12.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:12.949749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:23.170 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:23.169821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:23.223 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:23.223247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:33.438 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:33.438522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:33.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:33.449027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:43.706 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:43.706033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:43.715 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:43.715700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:53.951 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:53.950970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:33:53.961 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:33:53.961424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:04.201 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:04.200702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:04.211 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:04.211379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:14.443 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:14.443083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:14.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:14.453546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:24.734 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:24.734412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:24.747 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:24.745876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:34.985 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:34.984531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:34.996 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:34.995787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:45.288 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:45.287240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:45.298 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:45.297504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:55.606 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:55.606240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:34:55.655 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:34:55.614769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:05.836 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:05.836043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:05.849 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:05.849487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:16.067 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:16.067420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:16.081 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:16.081113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:26.316 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:26.315937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:26.420 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:26.330349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:36.610 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:36.610024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:36.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:36.621171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:46.837 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:46.837011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:46.888 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:46.887905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:57.106 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:57.106120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:35:57.205 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:35:57.118159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:07.366 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:07.365383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:07.382 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:07.378455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:17.628 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:17.627660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:17.704 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:17.704112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:27.924 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:27.923848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:27.938 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:27.937904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:31.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:31.682597 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T04:36:32.306 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:32.305122 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:36:32.293+0000 7f6c35ff2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:36:32.306 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:36:32.293+0000 7f6c35ff2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:36:32.306 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:36:32.293+0000 7f6c35ff2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:36:32.307 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:36:32.307 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:32.305144 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:36:32.293+0000 7f6c35ff2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:36:32.307 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:36:32.293+0000 7f6c35ff2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:36:32.307 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:36:32.293+0000 7f6c35ff2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:36:32.308 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:36:38.160 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:38.159935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:38.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:38.173759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:48.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:48.401923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:48.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:48.433116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:58.756 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:58.756451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:36:58.771 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:36:58.770993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:08.986 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:08.985591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:08.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:08.997665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:19.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:19.214830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:19.230 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:19.229306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:29.445 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:29.444738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:29.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:29.457477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:39.677 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:39.677532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:39.689 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:39.689694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:49.915 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:49.915263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:37:49.928 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:37:49.928637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:00.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:00.214896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:00.227 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:00.226919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:10.441 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:10.441035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:10.454 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:10.453773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:20.677 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:20.676817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:20.690 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:20.690054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:30.913 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:30.912927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:30.928 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:30.927896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:41.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:41.143761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:41.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:41.156677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:51.377 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:51.376183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:38:51.389 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:38:51.388370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:01.610 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:01.609623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:01.642 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:01.641737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:11.861 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:11.860918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:11.871 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:11.871324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:22.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:22.101318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:22.112 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:22.112294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:32.348 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:32.347872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:32.363 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:32.363592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:42.581 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:42.580346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:42.593 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:42.593053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:52.853 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:52.852526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:39:52.866 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:39:52.865724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:03.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:03.087552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:03.099 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:03.099265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:13.352 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:13.351553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:13.364 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:13.363933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:23.582 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:23.581910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:23.594 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:23.593585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:33.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:33.811996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:33.825 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:33.823316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:44.043 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:44.041039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:44.051 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:44.049920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:54.262 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:54.262024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:40:54.276 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:40:54.276151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:04.492 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:04.491775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:04.504 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:04.503833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:14.717 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:14.716843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:14.729 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:14.728860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:24.951 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:24.950411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:24.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:24.963006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:35.186 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:35.185716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:35.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:35.204107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:45.428 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:45.427924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:45.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:45.440034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:55.655 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:55.655309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:41:55.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:41:55.669004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:05.892 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:05.891980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:05.905 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:05.904448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:16.124 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:16.123638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:16.136 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:16.135889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:26.354 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:26.353863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:26.366 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:26.366261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:36.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:36.589977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:36.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:36.604314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:46.821 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:46.820543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:46.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:46.832017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:57.050 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:57.050173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:42:57.064 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:42:57.063737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:07.278 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:07.278299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:07.290 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:07.289862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:17.512 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:17.511626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:17.527 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:17.527018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:27.742 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:27.741740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:27.769 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:27.768857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:37.994 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:37.994065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:38.006 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:38.006563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:48.313 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:48.313373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:48.328 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:48.328564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:58.572 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:58.569028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:43:58.586 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:43:58.586248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:08.831 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:08.830999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:08.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:08.844186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:19.074 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:19.074282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:19.088 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:19.087834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:29.347 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:29.346773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:29.365 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:29.365399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:39.587 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:39.586749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:39.597 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:39.597320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:49.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:49.844489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:44:49.857 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:44:49.856693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:00.101 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:00.101464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:00.112 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:00.112185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:10.333 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:10.331991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:10.344 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:10.344933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:20.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:20.565347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:20.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:20.576997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:30.797 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:30.796592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:30.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:30.809064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:41.028 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:41.027745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:41.037 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:41.037780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:51.255 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:51.254347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:45:51.264 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:45:51.264199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:01.485 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:01.485381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:01.494 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:01.494704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:11.766 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:11.766334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:11.778 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:11.778125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:21.999 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:21.999334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:22.011 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:22.011803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:32.229 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:32.228777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:32.241 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:32.241315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:42.461 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:42.460468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:42.473 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:42.473232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:52.695 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:52.695099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:46:52.706 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:46:52.706972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:02.948 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:02.948543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:02.974 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:02.974756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:13.200 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:13.200099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:13.212 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:13.212740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:23.433 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:23.432888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:23.446 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:23.446486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:33.695 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:33.695730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:33.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:33.706954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:44.003 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:44.003252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:44.017 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:44.016830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:54.240 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:54.237987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:47:54.251 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:47:54.251237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:04.472 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:04.471946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:04.479 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:04.479971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:14.732 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:14.731673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:14.745 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:14.745558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:24.992 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:24.992187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:25.004 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:25.004654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:35.285 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:35.284318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:35.297 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:35.296698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:45.535 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:45.535214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:45.545 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:45.544237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:55.763 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:55.762880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:55.773 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:55.773990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:48:56.009 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:48:56.009960 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 04:48:55.998810055 +0000 UTC m=+12264.137349043 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T04:49:05.998 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:05.997929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:06.012 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:06.011969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:16.234 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:16.234368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:16.248 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:16.248283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:26.470 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:26.470379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:26.482 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:26.481589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:36.703 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:36.702846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:36.711 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:36.712038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:46.926 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:46.926386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:46.939 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:46.938711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:56.971 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:56.971004 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 04:49:56.963852058 +0000 UTC m=+12325.102390993 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T04:49:57.157 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:57.156965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:49:57.168 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:49:57.168256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:07.390 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:07.390137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:07.401 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:07.401536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:17.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:17.621002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:17.629 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:17.629065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:27.853 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:27.853233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:27.865 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:27.865735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:38.100 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:38.100055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:38.168 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:38.168077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:48.576 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:48.576144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:48.588 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:48.588886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:58.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:58.807731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:50:58.817 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:50:58.816841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:09.036 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:09.036740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:09.047 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:09.048053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:19.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:19.267743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:19.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:19.281884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:29.498 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:29.498002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:29.512 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:29.512403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:39.731 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:39.731344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:39.743 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:39.743122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:49.963 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:49.963532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:51:49.975 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:51:49.975964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:00.203 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:00.202172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:00.214 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:00.213700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:10.436 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:10.435939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:10.448 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:10.448269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:20.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:20.664673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:20.676 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:20.676422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:30.892 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:30.891883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:30.904 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:30.904574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:41.125 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:41.124885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:41.138 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:41.138946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:51.384 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:51.382945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:52:51.395 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:52:51.396003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:01.666 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:01.666236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:01.678 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:01.678161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:11.919 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:11.919497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:11.933 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:11.933861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:12.317 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:12.317001 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T04:53:12.940 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:12.939860 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:53:12.931+0000 7f6536b6a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:53:12.941 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:53:12.931+0000 7f6536b6a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:53:12.941 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:53:12.931+0000 7f6536b6a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:53:12.941 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:53:12.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:12.939889 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:53:12.931+0000 7f6536b6a340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:53:12.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:53:12.931+0000 7f6536b6a340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:53:12.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T04:53:12.931+0000 7f6536b6a340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:53:12.942 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:53:22.158 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:22.158251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:22.171 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:22.171277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:32.390 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:32.390644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:32.407 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:32.405411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:42.662 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:42.662420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:42.674 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:42.674915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:52.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:52.897307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:53:52.911 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:53:52.911527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:03.164 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:03.164210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:03.197 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:03.197325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:13.430 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:13.429018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:13.440 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:13.441157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:23.689 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:23.689274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:23.698 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:23.698420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:33.942 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:33.942951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:33.953 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:33.954047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:44.173 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:44.173139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:44.187 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:44.186462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:54.418 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:54.418452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:54:54.430 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:54:54.430405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:04.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:04.647088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:04.655 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:04.655964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:14.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:14.879645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:14.936 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:14.936440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:25.156 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:25.156490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:25.175 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:25.175871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:35.402 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:35.402618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:35.415 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:35.415169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:45.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:45.635567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:45.650 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:45.648788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:55.869 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:55.868737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:55:55.908 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:55:55.908185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:06.144 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:06.144328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:06.156 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:06.156692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:16.375 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:16.375090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:16.388 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:16.388268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:26.604 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:26.603640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:26.624 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:26.616499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:36.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:36.867314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:36.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:36.883332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:47.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:47.123411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:47.131 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:47.131904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:57.380 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:57.380750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:56:57.391 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:56:57.392067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:07.654 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:07.654703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:07.669 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:07.669161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:17.885 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:17.884924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:17.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:17.897890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:28.133 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:28.133587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:28.147 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:28.147370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:38.369 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:38.369679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:38.421 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:38.421032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:48.644 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:48.644500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:48.658 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:48.658527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:58.879 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:58.879573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:57:58.891 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:57:58.891258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:09.111 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:09.111117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:09.122 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:09.122834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:19.346 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:19.346602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:19.355 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:19.355413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:29.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:29.574789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:29.588 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:29.587612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:39.806 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:39.806476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:39.822 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:39.822204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:50.045 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:50.044883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:58:50.059 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:58:50.058553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:00.276 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:00.276494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:00.289 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:00.289618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:10.548 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:10.548731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:10.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:10.565421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:20.863 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:20.863811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:20.875 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:20.875633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:31.091 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:31.091433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:31.103 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:31.103212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:41.327 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:41.327094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:41.341 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:41.340853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:51.580 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:51.580485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T04:59:51.590 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 04:59:51.590505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:01.808 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:01.808323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:01.820 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:01.820669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:12.037 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:12.035784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:12.051 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:12.051857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:22.272 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:22.271977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:22.285 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:22.284776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:32.505 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:32.504861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:32.519 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:32.519454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:42.742 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:42.742562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:42.753 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:42.751404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:52.987 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:52.987296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:00:53.001 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:00:53.000051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:03.222 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:03.221478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:03.236 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:03.230287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:13.453 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:13.453215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:13.466 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:13.466785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:23.687 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:23.687557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:23.700 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:23.699720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:33.927 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:33.927107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:33.940 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:33.940696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:44.160 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:44.160058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:44.174 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:44.174197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:44.174 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:44.174241 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:54.398 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:54.398265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:01:54.410 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:01:54.410579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:04.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:04.627032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:04.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:04.635672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:14.854 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:14.853900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:14.866 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:14.866025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:25.125 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:25.125116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:25.137 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:25.138041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:35.406 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:35.406390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:35.419 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:35.419876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:45.671 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:45.671193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:45.683 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:45.683945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:55.904 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:55.904240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:02:55.916 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:02:55.916466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:06.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:06.139537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:06.154 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:06.153450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:16.391 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:16.391602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:16.400 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:16.401211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:26.621 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:26.620692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:26.630 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:26.630961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:36.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:36.897000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:36.914 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:36.914145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:47.137 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:47.137169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:47.150 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:47.150510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:57.386 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:57.385591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:03:57.397 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:03:57.397547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:07.634 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:07.634058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:07.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:07.647030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:17.889 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:17.889670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:17.903 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:17.903145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:28.172 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:28.171877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:28.184 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:28.184208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:38.408 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:38.408552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:38.420 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:38.420942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:48.639 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:48.639856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:48.685 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:48.684515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:58.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:58.897600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:04:58.912 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:04:58.912741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:09.134 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:09.134286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:09.148 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:09.149140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:19.368 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:19.368108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:19.377 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:19.377691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:29.598 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:29.598672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:29.609 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:29.609534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:39.831 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:39.830971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:39.844 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:39.843995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:50.070 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:50.069766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:05:50.081 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:05:50.081961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:00.303 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:00.303428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:00.315 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:00.315572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:10.567 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:10.566332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:10.575 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:10.575424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:10.920 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:10.920718 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 05:06:10.910284631 +0000 UTC m=+13299.048823609 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T05:06:20.805 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:20.805640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:21.009 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:20.815575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:31.079 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:31.079678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:31.089 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:31.089150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:41.308 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:41.308623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:41.324 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:41.325146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:51.546 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:51.546242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:06:52.003 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:06:51.558612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:01.780 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:01.780156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:01.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:01.792682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:12.051 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:12.050376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:12.061 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:12.061287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:22.279 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:22.279791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:22.294 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:22.293845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:32.538 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:32.537920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:32.546 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:32.547032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:42.768 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:42.768390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:42.782 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:42.782657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:53.003 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:53.002823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:07:53.015 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:07:53.015145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:03.235 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:03.235544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:03.247 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:03.248049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:13.479 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:13.479050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:13.487 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:13.487301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:23.755 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:23.755272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:23.765 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:23.765204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:34.009 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:34.009308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:34.022 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:34.022839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:44.243 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:44.243036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:44.257 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:44.257590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:54.478 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:54.477355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:08:54.489 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:08:54.489615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:04.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:04.706951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:04.719 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:04.719696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:14.944 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:14.943878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:14.958 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:14.959035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:25.181 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:25.181779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:25.193 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:25.193228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:35.415 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:35.415019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:35.430 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:35.429932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:45.652 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:45.652306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:45.667 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:45.667682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:52.947 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:52.947425 I | op-mon: parsing mon endpoints: a=10.107.13.124:6789 2022-03-10T05:09:53.551 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:53.551040 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T05:09:53.541+0000 7fdca0061340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T05:09:53.552 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T05:09:53.541+0000 7fdca0061340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T05:09:53.552 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T05:09:53.541+0000 7fdca0061340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T05:09:53.552 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T05:09:53.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:53.551075 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T05:09:53.541+0000 7fdca0061340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T05:09:53.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T05:09:53.541+0000 7fdca0061340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T05:09:53.553 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10T05:09:53.541+0000 7fdca0061340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T05:09:53.554 INFO:tasks.rook.operator.smithi186.stdout:couldn't init storage provider. : exit status 5 2022-03-10T05:09:55.888 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:55.888371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:09:55.897 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:09:55.897440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:06.121 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:06.120719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:06.137 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:06.137296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:16.392 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:16.392714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:16.406 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:16.406597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:26.635 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:26.635332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:26.644 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:26.644329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:36.864 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:36.864356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:36.876 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:36.876616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:47.099 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:47.099136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:47.111 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:47.111521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:57.332 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:57.332492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:10:57.343 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:10:57.343595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:07.566 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:07.566552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:07.578 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:07.578234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:17.797 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:17.796975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:17.809 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:17.809554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:28.026 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:28.026374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:28.036 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:28.036927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:38.255 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:38.255169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:38.321 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:38.321621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:48.541 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:48.541680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:48.552 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:48.552827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:58.786 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:58.786693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:11:58.800 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:11:58.800178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:09.062 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:09.062368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:09.076 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:09.076398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:19.295 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:19.294820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:19.308 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:19.308597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:29.527 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:29.527704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:29.538 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:29.538328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:39.758 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:39.758647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:39.768 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:39.769169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:50.053 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:50.052881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:12:50.065 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:12:50.064995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:00.284 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:00.284393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:00.297 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:00.297683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:10.512 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:10.512361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:10.524 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:10.524330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:20.741 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:20.741707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:20.751 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:20.751651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:30.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:30.972613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:30.982 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:30.983356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:41.204 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:41.204164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:41.215 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:41.215596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:51.442 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:51.442254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:13:51.455 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:13:51.455455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:01.707 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:01.707059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:01.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:01.721125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:11.938 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:11.938107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:11.950 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:11.950354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:22.168 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:22.168742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:22.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:22.182909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:32.405 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:32.404873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:32.428 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:32.421863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:42.671 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:42.670995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:42.682 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:42.682971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:52.906 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:52.906000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:14:52.917 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:14:52.918293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:03.143 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:03.143172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:03.152 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:03.152446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:13.437 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:13.437871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:13.449 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:13.449959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:23.667 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:23.667911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:23.679 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:23.679369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:33.912 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:33.912634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:33.923 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:33.924112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:44.139 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:44.139565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:44.151 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:44.151042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:54.368 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:54.368716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:15:54.379 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:15:54.379246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:04.598 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:04.598083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:04.610 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:04.610802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:14.854 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:14.854324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:14.867 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:14.867126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:25.081 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:25.081594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:25.095 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:25.094592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:35.322 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:35.322664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:35.332 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:35.332823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:45.565 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:45.565428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:45.574 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:45.574420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:55.792 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:55.791723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:16:55.804 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:16:55.804766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:06.052 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:06.051980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:06.061 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:06.061379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:16.281 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:16.281472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:16.295 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:16.295937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:26.525 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:26.526025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:26.540 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:26.538973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:36.959 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:36.959404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:36.970 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:36.970815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:47.198 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:47.198043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:47.216 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:47.216348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:57.434 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:57.434287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:17:57.448 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:17:57.449072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:07.709 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:07.709100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:07.720 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:07.720698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:17.943 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:17.943547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:17.952 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:17.953018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:28.183 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:28.183387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:28.195 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:28.196010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:38.416 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:38.416737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:38.429 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:38.429197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:48.647 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:48.647455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:48.664 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:48.665339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:58.883 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:58.883456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:18:58.892 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:18:58.892432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:09.110 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:09.110787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:09.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:09.123564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:19.359 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:19.359753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:19.374 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:19.374896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:29.618 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:29.616689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:29.627 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:29.628482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:39.962 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:39.962497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:39.972 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:39.973127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:50.247 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:50.247364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:19:50.259 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:19:50.259631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:00.479 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:00.479001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:00.491 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:00.491489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:10.802 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:10.802026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:10.813 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:10.813445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:21.033 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:21.033776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:21.046 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:21.047092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:31.267 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:31.267861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:31.280 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:31.280323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:41.501 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:41.501876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:41.534 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:41.534357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:51.756 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:51.755781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:20:51.767 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:20:51.767412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:01.988 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:01.988250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:02.001 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:02.001649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:12.219 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:12.219621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:12.232 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:12.232997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:22.454 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:22.454244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:22.465 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:22.465919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:32.693 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:32.693155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:32.703 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:32.704333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:42.964 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:42.964231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:42.977 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:42.978132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:53.197 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:53.197076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:21:53.208 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:21:53.208268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:03.429 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:03.429753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:03.441 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:03.442089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:13.662 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:13.662837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:13.675 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:13.675835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:23.893 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:23.893205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:23.900 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:23.901248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:34.123 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:34.123138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:34.133 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:34.133981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:44.357 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:44.357268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:44.368 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:44.368815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:54.585 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:54.585292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:22:54.596 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:22:54.597464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:04.817 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:04.817291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:04.828 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:04.829071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:15.047 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:15.047125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:15.058 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:15.059157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:25.282 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:25.282211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:25.293 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:25.294333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:25.681 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:25.682155 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 05:23:25.665944439 +0000 UTC m=+14333.804483381 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T05:23:35.513 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:35.513561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:35.530 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:35.531187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:45.784 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:45.784711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:45.795 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:45.795857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:56.037 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:56.037515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:23:56.048 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:23:56.048881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:24:06.293 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:24:06.293144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:24:06.305 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:24:06.306037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:24:16.562 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:24:16.562525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:24:16.571 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:24:16.571455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:24:26.721 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:24:26.721882 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 05:24:26.684270436 +0000 UTC m=+14394.822809436 LastTransitionTime:2022-03-10 01:26:42 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T05:24:26.834 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:24:26.834484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T05:24:26.847 INFO:tasks.rook.operator.smithi186.stdout:2022-03-10 05:24:26.847347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-10T07:44:46.975 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-10T07:44:47.000 DEBUG:teuthology.task.console_log:Killing console logger for smithi186 2022-03-10T07:44:47.001 DEBUG:teuthology.exit:Finished running handlers