2022-03-20T14:44:33.050 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-20T14:44:33.052 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/1.7.2} --archive /home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748896 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748896/orig.config.yaml 2022-03-20T14:44:33.088 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-20T14:44:33.151 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748896 branch: wip-yuri6-testing-2022-03-18-1104 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/1.7.2} email: null first_in_suite: false job_id: '6748896' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri6-testing-2022-03-18-1104 ceph: conf: global: osd crush chooseleaf type: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e spec: mon: allowMultiplePerNode: false network: provider: host ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e kernel: hwe: true kubeadm: version: '1.21' rook: rook_branch: v1.7.2 rook_image: rook/ceph:v1.7.2 workunit: branch: wip-yuri6-testing-2022-03-18-1104 sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri6-testing-2022-03-18-1104 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e targets: smithi134.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDzV8i2zirpVD7daMH5hFGQjM3O2ANkQaF1pAUeajjBj9goSood/TMT8PEXDn6AXhtNe2ym4KB+Io9E+IMe+vN8kWzTHkoVdMhlGW6D2yV/DbMyD7qaBSrffABcO+cRkoCaAVHLYyZCAb7NZUKoiLUBZNpfQNTEDlMLwGxJAnwvkpZie6xVA+xnjW0FGYedMyqRZEtmohOAM/U5q2eL2aqWlfihAYXXJ6uS8fQQFRO0Vl5h9t70yvu884eSGtRxwi7k+BEWDDOw+fxbARlVL6URCBcXsMjXmnNg1SzCebsHIqae52ysdwBrvGLbNCUunH7RK7gzMUe9Wvyf5YAKdgIuW9QiXKKFwoUw6b10HEmUhtkGGrVdhXLLGnWHiWPXi430+ndib+aIPFjet4RBQkXzxV26jo28IqI9eyzNq4WxcFVERC6xgWasGtEzXyiIsy2nx6Edui0HYgeXMHmY6a3ffz+NvoO18ApEaqVBCjh3wlXVdND2kwKllC0z4LZTa0= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - install: host.a: null - radosbench: clients: - client.a - rook.shell: commands: - 'ceph orch host label add `hostname` foo ceph orch host ls | grep foo ceph orch host label rm `hostname` foo ceph orch host ls | grep -v foo ' - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: a4e18662e5de936eb0ac089e531b933ecbcadb0a timestamp: 2022-03-20_14:28:38 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-20T14:44:33.152 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa; will attempt to use it 2022-03-20T14:44:33.153 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks 2022-03-20T14:44:33.153 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-20T14:44:33.155 INFO:teuthology.task.internal:Checking packages... 2022-03-20T14:44:33.176 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e' 2022-03-20T14:44:33.177 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-20T14:44:33.177 INFO:teuthology.packaging:ref: None 2022-03-20T14:44:33.178 INFO:teuthology.packaging:tag: None 2022-03-20T14:44:33.178 INFO:teuthology.packaging:branch: wip-yuri6-testing-2022-03-18-1104 2022-03-20T14:44:33.178 INFO:teuthology.packaging:sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-20T14:44:33.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri6-testing-2022-03-18-1104 2022-03-20T14:44:33.333 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11187-ge92561bf-1focal 2022-03-20T14:44:33.334 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-20T14:44:33.342 INFO:teuthology.task.internal:no buildpackages task found 2022-03-20T14:44:33.343 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-20T14:44:33.376 INFO:teuthology.task.internal:Saving configuration 2022-03-20T14:44:33.389 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-20T14:44:33.400 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-20T14:44:33.423 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748896', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-20 14:39:13.118537', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFnvhZoK1JbAn4ChhXrQ4Xv/qscScEC9tuIsPrMLDSHD'} 2022-03-20T14:44:33.423 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-20T14:44:33.431 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-20T14:44:33.431 INFO:teuthology.run_tasks:Running task console_log... 2022-03-20T14:44:33.489 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f232a0a7c80>, signals=[15]) 2022-03-20T14:44:33.490 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-20T14:44:33.497 INFO:teuthology.task.internal:Opening connections... 2022-03-20T14:44:33.498 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2022-03-20T14:44:33.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:44:33.545 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-20T14:44:33.553 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2022-03-20T14:44:33.669 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2022-03-20T14:44:33.669 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2022-03-20T14:44:33.719 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="Ubuntu" 2022-03-20T14:44:33.719 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-20T14:44:33.719 INFO:teuthology.orchestra.run.smithi134.stdout:ID=ubuntu 2022-03-20T14:44:33.719 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE=debian 2022-03-20T14:44:33.720 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-20T14:44:33.720 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="20.04" 2022-03-20T14:44:33.720 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-20T14:44:33.720 INFO:teuthology.orchestra.run.smithi134.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-20T14:44:33.721 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-20T14:44:33.721 INFO:teuthology.orchestra.run.smithi134.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-20T14:44:33.721 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_CODENAME=focal 2022-03-20T14:44:33.721 INFO:teuthology.orchestra.run.smithi134.stdout:UBUNTU_CODENAME=focal 2022-03-20T14:44:33.722 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2022-03-20T14:44:33.741 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-20T14:44:33.752 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-20T14:44:33.793 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-20T14:44:33.793 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2022-03-20T14:44:33.799 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-20T14:44:33.818 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-20T14:44:33.819 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2022-03-20T14:44:33.849 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-20T14:44:33.879 INFO:teuthology.run_tasks:Running task kernel... 2022-03-20T14:44:33.895 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-20T14:44:33.895 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-20T14:44:33.896 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}} 2022-03-20T14:44:33.896 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-20T14:44:33.896 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-20T14:44:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-20T14:44:33.903 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2022-03-20T14:44:33.909 INFO:teuthology.orchestra.run.smithi134.stdout:5.4.0-97-generic 2022-03-20T14:44:33.910 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.4.0-97-generic 2022-03-20T14:44:33.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get clean 2022-03-20T14:44:33.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get update 2022-03-20T14:44:34.142 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:44:34.151 INFO:teuthology.orchestra.run.smithi134.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-20T14:44:34.223 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-20T14:44:34.242 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-20T14:44:34.535 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-20T14:44:34.623 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-20T14:44:34.654 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-20T14:44:34.657 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-20T14:44:34.658 INFO:teuthology.orchestra.run.smithi134.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-20T14:44:34.658 INFO:teuthology.orchestra.run.smithi134.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-20T14:44:34.668 INFO:teuthology.orchestra.run.smithi134.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-20T14:44:34.669 INFO:teuthology.orchestra.run.smithi134.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-20T14:44:34.670 INFO:teuthology.orchestra.run.smithi134.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-20T14:44:34.676 INFO:teuthology.orchestra.run.smithi134.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-20T14:44:34.686 INFO:teuthology.orchestra.run.smithi134.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-20T14:44:34.688 INFO:teuthology.orchestra.run.smithi134.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-20T14:44:34.689 INFO:teuthology.orchestra.run.smithi134.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-20T14:44:34.690 INFO:teuthology.orchestra.run.smithi134.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-20T14:44:34.691 INFO:teuthology.orchestra.run.smithi134.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-20T14:44:34.691 INFO:teuthology.orchestra.run.smithi134.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-20T14:44:34.691 INFO:teuthology.orchestra.run.smithi134.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-20T14:44:34.699 INFO:teuthology.orchestra.run.smithi134.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-20T14:44:34.700 INFO:teuthology.orchestra.run.smithi134.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-20T14:44:34.719 INFO:teuthology.orchestra.run.smithi134.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-20T14:44:34.720 INFO:teuthology.orchestra.run.smithi134.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-20T14:44:34.721 INFO:teuthology.orchestra.run.smithi134.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-20T14:44:34.721 INFO:teuthology.orchestra.run.smithi134.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-20T14:44:34.855 INFO:teuthology.orchestra.run.smithi134.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-20T14:44:35.107 INFO:teuthology.orchestra.run.smithi134.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-20T14:44:35.260 INFO:teuthology.orchestra.run.smithi134.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-20T14:44:35.275 INFO:teuthology.orchestra.run.smithi134.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-20T14:44:35.275 INFO:teuthology.orchestra.run.smithi134.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-20T14:44:35.312 INFO:teuthology.orchestra.run.smithi134.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-20T14:44:35.313 INFO:teuthology.orchestra.run.smithi134.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-20T14:44:35.317 INFO:teuthology.orchestra.run.smithi134.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-20T14:44:35.317 INFO:teuthology.orchestra.run.smithi134.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-20T14:44:35.342 INFO:teuthology.orchestra.run.smithi134.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-20T14:44:35.363 INFO:teuthology.orchestra.run.smithi134.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-20T14:44:35.366 INFO:teuthology.orchestra.run.smithi134.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-20T14:44:35.422 INFO:teuthology.orchestra.run.smithi134.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-20T14:44:35.423 INFO:teuthology.orchestra.run.smithi134.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-20T14:44:35.423 INFO:teuthology.orchestra.run.smithi134.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-20T14:44:35.424 INFO:teuthology.orchestra.run.smithi134.stdout:Get:43 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-20T14:44:37.866 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 10.2 MB in 2s (6,152 kB/s) 2022-03-20T14:44:39.024 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:44:39.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:44:39.125 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:44:39.298 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2022-03-20T14:44:39.299 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2022-03-20T14:44:39.440 INFO:teuthology.orchestra.run.smithi134.stdout:The following additional packages will be installed: 2022-03-20T14:44:39.440 INFO:teuthology.orchestra.run.smithi134.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:44:39.440 INFO:teuthology.orchestra.run.smithi134.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-20T14:44:39.440 INFO:teuthology.orchestra.run.smithi134.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-20T14:44:39.441 INFO:teuthology.orchestra.run.smithi134.stdout:Suggested packages: 2022-03-20T14:44:39.441 INFO:teuthology.orchestra.run.smithi134.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-20T14:44:39.441 INFO:teuthology.orchestra.run.smithi134.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-20T14:44:39.490 INFO:teuthology.orchestra.run.smithi134.stdout:The following NEW packages will be installed: 2022-03-20T14:44:39.491 INFO:teuthology.orchestra.run.smithi134.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:44:39.492 INFO:teuthology.orchestra.run.smithi134.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-20T14:44:39.492 INFO:teuthology.orchestra.run.smithi134.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-20T14:44:39.492 INFO:teuthology.orchestra.run.smithi134.stdout: thermald upower usbmuxd 2022-03-20T14:44:39.547 INFO:teuthology.orchestra.run.smithi134.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:44:39.548 INFO:teuthology.orchestra.run.smithi134.stdout:Need to get 80.1 MB of archives. 2022-03-20T14:44:39.548 INFO:teuthology.orchestra.run.smithi134.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-20T14:44:39.548 INFO:teuthology.orchestra.run.smithi134.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-20T14:44:39.613 INFO:teuthology.orchestra.run.smithi134.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-20T14:44:39.622 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-20T14:44:39.628 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-20T14:44:39.645 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-20T14:44:39.652 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-20T14:44:39.912 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-20T14:44:40.000 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-20T14:44:40.673 INFO:teuthology.orchestra.run.smithi134.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-20T14:44:40.673 INFO:teuthology.orchestra.run.smithi134.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-20T14:44:40.673 INFO:teuthology.orchestra.run.smithi134.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-20T14:44:40.677 INFO:teuthology.orchestra.run.smithi134.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-20T14:44:40.679 INFO:teuthology.orchestra.run.smithi134.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-20T14:44:41.081 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 80.1 MB in 1s (67.8 MB/s) 2022-03-20T14:44:41.303 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-20T14:44:42.807 INFO:teuthology.orchestra.run.smithi134.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-20T14:44:42.811 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-20T14:44:42.949 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:44:43.336 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-20T14:44:43.351 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-20T14:44:43.393 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:44:43.672 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-20T14:44:43.687 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-20T14:44:43.721 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:44:44.049 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-20T14:44:44.065 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-20T14:44:44.108 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:44:44.477 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-20T14:44:44.489 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-20T14:44:44.527 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:44:44.830 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-20T14:44:44.844 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:44:44.888 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:44:47.127 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-20T14:44:47.143 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:44:47.235 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:44:47.749 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-20T14:44:47.762 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:44:47.799 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:44:53.780 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-20T14:44:53.798 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-20T14:44:53.863 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:44:54.149 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-20T14:44:54.163 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-20T14:44:54.198 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:44:54.579 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package thermald. 2022-03-20T14:44:54.594 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-20T14:44:54.637 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:44:55.178 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package upower. 2022-03-20T14:44:55.191 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-20T14:44:55.246 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-20T14:44:55.661 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package usbmuxd. 2022-03-20T14:44:55.675 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-20T14:44:55.719 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:44:56.002 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:44:56.142 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:44:56.269 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:44:56.795 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:44:56.896 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:44:57.022 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:44:57.181 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-20T14:44:57.635 INFO:teuthology.orchestra.run.smithi134.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-20T14:44:57.707 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:44:57.891 INFO:teuthology.orchestra.run.smithi134.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-20T14:44:57.896 INFO:teuthology.orchestra.run.smithi134.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-20T14:44:57.897 INFO:teuthology.orchestra.run.smithi134.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-20T14:44:58.383 INFO:teuthology.orchestra.run.smithi134.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-20T14:44:58.606 INFO:teuthology.orchestra.run.smithi134.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-20T14:44:58.646 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:44:58.965 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:44:58.966 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:44:59.433 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:45:00.686 INFO:teuthology.orchestra.run.smithi134.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-20T14:45:00.687 INFO:teuthology.orchestra.run.smithi134.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-20T14:45:00.843 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:45:02.178 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:45:03.471 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:45:03.597 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-20T14:45:07.919 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:45:08.596 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-20T14:45:08.696 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:45:08.882 INFO:teuthology.orchestra.run.smithi134.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-20T14:45:08.883 INFO:teuthology.orchestra.run.smithi134.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:45:32.111 INFO:teuthology.orchestra.run.smithi134.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-20T14:45:32.111 INFO:teuthology.orchestra.run.smithi134.stdout:Sourcing file `/etc/default/grub' 2022-03-20T14:45:32.116 INFO:teuthology.orchestra.run.smithi134.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:45:32.181 INFO:teuthology.orchestra.run.smithi134.stdout:Generating grub configuration file ... 2022-03-20T14:45:32.537 INFO:teuthology.orchestra.run.smithi134.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:45:32.574 INFO:teuthology.orchestra.run.smithi134.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:45:33.068 INFO:teuthology.orchestra.run.smithi134.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:45:33.080 INFO:teuthology.orchestra.run.smithi134.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:45:35.756 INFO:teuthology.orchestra.run.smithi134.stdout:done 2022-03-20T14:45:36.127 DEBUG:teuthology.orchestra.run.smithi134:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:45:36.157 INFO:teuthology.orchestra.run.smithi134.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:45:36.157 INFO:teuthology.orchestra.run.smithi134.stdout:Status: install ok installed 2022-03-20T14:45:36.157 INFO:teuthology.orchestra.run.smithi134.stdout:Priority: optional 2022-03-20T14:45:36.157 INFO:teuthology.orchestra.run.smithi134.stdout:Section: kernel 2022-03-20T14:45:36.158 INFO:teuthology.orchestra.run.smithi134.stdout:Installed-Size: 19 2022-03-20T14:45:36.158 INFO:teuthology.orchestra.run.smithi134.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:45:36.158 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: amd64 2022-03-20T14:45:36.158 INFO:teuthology.orchestra.run.smithi134.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:45:36.159 INFO:teuthology.orchestra.run.smithi134.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:45:36.159 INFO:teuthology.orchestra.run.smithi134.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:45:36.159 INFO:teuthology.orchestra.run.smithi134.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:45:36.160 INFO:teuthology.orchestra.run.smithi134.stdout:Recommends: thermald 2022-03-20T14:45:36.160 INFO:teuthology.orchestra.run.smithi134.stdout:Description: Generic Linux kernel image 2022-03-20T14:45:36.160 INFO:teuthology.orchestra.run.smithi134.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:45:36.160 INFO:teuthology.orchestra.run.smithi134.stdout: available. 2022-03-20T14:45:36.161 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-20T14:45:36.161 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-20T14:45:36.161 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-20T14:45:36.161 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-20T14:45:36.162 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2022-03-20T14:45:36.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get clean 2022-03-20T14:45:36.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get update 2022-03-20T14:45:36.355 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:45:36.356 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:45:36.364 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:45:36.384 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:45:37.905 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:45:37.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:45:37.993 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:45:38.167 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2022-03-20T14:45:38.168 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2022-03-20T14:45:38.352 INFO:teuthology.orchestra.run.smithi134.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-20T14:45:38.352 INFO:teuthology.orchestra.run.smithi134.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:45:38.354 DEBUG:teuthology.orchestra.run.smithi134:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:45:38.375 INFO:teuthology.orchestra.run.smithi134.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:45:38.375 INFO:teuthology.orchestra.run.smithi134.stdout:Status: install ok installed 2022-03-20T14:45:38.376 INFO:teuthology.orchestra.run.smithi134.stdout:Priority: optional 2022-03-20T14:45:38.376 INFO:teuthology.orchestra.run.smithi134.stdout:Section: kernel 2022-03-20T14:45:38.376 INFO:teuthology.orchestra.run.smithi134.stdout:Installed-Size: 19 2022-03-20T14:45:38.376 INFO:teuthology.orchestra.run.smithi134.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:45:38.377 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: amd64 2022-03-20T14:45:38.377 INFO:teuthology.orchestra.run.smithi134.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:45:38.377 INFO:teuthology.orchestra.run.smithi134.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:45:38.377 INFO:teuthology.orchestra.run.smithi134.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:45:38.378 INFO:teuthology.orchestra.run.smithi134.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:45:38.378 INFO:teuthology.orchestra.run.smithi134.stdout:Recommends: thermald 2022-03-20T14:45:38.378 INFO:teuthology.orchestra.run.smithi134.stdout:Description: Generic Linux kernel image 2022-03-20T14:45:38.378 INFO:teuthology.orchestra.run.smithi134.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:45:38.379 INFO:teuthology.orchestra.run.smithi134.stdout: available. 2022-03-20T14:45:38.379 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2022-03-20T14:45:38.386 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.8j4oX4hTBH 2022-03-20T14:45:38.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub/grub.cfg /tmp/tmp.8j4oX4hTBH 2022-03-20T14:45:38.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.8j4oX4hTBH 2022-03-20T14:45:38.469 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.8j4oX4hTBH is 8KB 2022-03-20T14:45:38.483 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.8j4oX4hTBH 2022-03-20T14:45:38.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-20T14:45:38.546 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:45:38.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-20T14:45:38.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-20T14:45:38.578 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-20T14:45:38.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo update-grub 2022-03-20T14:45:39.830 INFO:teuthology.orchestra.run.smithi134.stderr:Sourcing file `/etc/default/grub' 2022-03-20T14:45:39.835 INFO:teuthology.orchestra.run.smithi134.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:45:39.846 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2022-03-20T14:45:40.159 INFO:teuthology.orchestra.run.smithi134.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:45:40.171 INFO:teuthology.orchestra.run.smithi134.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:45:40.625 INFO:teuthology.orchestra.run.smithi134.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:45:40.636 INFO:teuthology.orchestra.run.smithi134.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:45:42.311 INFO:teuthology.orchestra.run.smithi134.stderr:done 2022-03-20T14:45:42.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2022-03-20T14:46:12.346 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-20T14:46:12.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2022-03-20T14:46:12.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:46:30.859 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2022-03-20T14:47:00.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2022-03-20T14:47:00.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:47:19.247 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2022-03-20T14:47:49.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2022-03-20T14:47:49.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:47:49.564 DEBUG:teuthology.orchestra.run.smithi134:> true 2022-03-20T14:47:50.018 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2022-03-20T14:47:50.019 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-20T14:47:50.019 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2022-03-20T14:47:50.023 INFO:teuthology.orchestra.run.smithi134.stdout:5.13.0-35-generic 2022-03-20T14:47:50.024 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-20T14:47:50.024 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-20T14:47:50.024 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-20T14:47:51.026 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-20T14:47:51.026 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-20T14:47:51.079 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2022-03-20T14:47:51.119 DEBUG:teuthology.parallel:result is None 2022-03-20T14:47:51.119 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-20T14:47:51.128 INFO:teuthology.task.internal:Creating test directory... 2022-03-20T14:47:51.129 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-20T14:47:51.137 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-20T14:47:51.182 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-20T14:47:51.212 INFO:teuthology.task.internal:Creating archive directory... 2022-03-20T14:47:51.214 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-20T14:47:51.234 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-20T14:47:51.248 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-20T14:47:51.248 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-20T14:47:51.294 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T14:47:51.303 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T14:47:51.305 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-20T14:47:51.316 INFO:teuthology.task.internal:Configuring sudo... 2022-03-20T14:47:51.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-20T14:47:51.331 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-20T14:47:51.348 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-20T14:47:51.349 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-20T14:47:51.383 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-20T14:47:51.388 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-20T14:47:51.394 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:47:51.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-20T14:47:51.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2022-03-20T14:47:51.458 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-20T14:47:51.473 INFO:teuthology.task.internal:Starting timer... 2022-03-20T14:47:51.473 INFO:teuthology.run_tasks:Running task pcp... 2022-03-20T14:47:51.507 INFO:teuthology.run_tasks:Running task selinux... 2022-03-20T14:47:51.539 DEBUG:teuthology.task.selinux:Excluding smithi134: OS 'ubuntu' does not support SELinux 2022-03-20T14:47:51.539 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-20T14:47:51.539 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-20T14:47:51.540 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-20T14:47:51.540 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-20T14:47:51.580 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-03-20T14:47:51.587 INFO:teuthology.repo_utils:Fetching master from origin 2022-03-20T14:47:51.620 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-20T14:47:51.638 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-20T14:47:51.640 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-20T14:51:04.587 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2022-03-20T14:51:04.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2022-03-20T14:51:04.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:51:04.643 DEBUG:teuthology.orchestra.run.smithi134:> true 2022-03-20T14:51:04.760 INFO:teuthology.run_tasks:Running task clock... 2022-03-20T14:51:04.774 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-20T14:51:04.775 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-20T14:51:04.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-20T14:51:04.809 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-20T14:51:04.809 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: Command line: ntpd -gq 2022-03-20T14:51:04.810 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: proto: precision = 0.074 usec (-24) 2022-03-20T14:51:04.810 INFO:teuthology.orchestra.run.smithi134.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T14:51:04.811 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T14:51:04.811 INFO:teuthology.orchestra.run.smithi134.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-20T14:51:04.812 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: restrict ::: KOD does nothing without LIMITED. 2022-03-20T14:51:04.813 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: Listen and drop on 0 v6wildcard [::]:123 2022-03-20T14:51:04.813 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-20T14:51:04.813 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: Listen normally on 2 lo 127.0.0.1:123 2022-03-20T14:51:04.813 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: Listen normally on 3 enp3s0f1 172.21.15.134:123 2022-03-20T14:51:04.814 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: Listen normally on 4 lo [::1]:123 2022-03-20T14:51:04.814 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6de9%5]:123 2022-03-20T14:51:04.814 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:04 ntpd[9502]: Listening on routing socket on fd #22 for interface updates 2022-03-20T14:51:05.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:05.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:05.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:05.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:05.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:05.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:05.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:05.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:05.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:05.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:05.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:05.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:05 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:07.810 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:07.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:07.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:07.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:07.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:07.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:07.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:07.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:07.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:07.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:07.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:07.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:07 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:08.810 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:08 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:08.810 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:08 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:08.810 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:08 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:08.810 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:08 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:08.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:08 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:08.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:08 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:09.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:09.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:09.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:09.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:09.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:09.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:09.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:09.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:09.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:09.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:09.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:09.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:09.814 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:09.814 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:09.814 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:09.814 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:09.815 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:09.815 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:09 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:10 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:10 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:10 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:10 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:10 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:10 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:11.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:11.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:11.811 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:11.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:11.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:11.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:11.812 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:11.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:11.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:11.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:11.813 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:51:11.814 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:51:11.814 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:51:11.814 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:51:11.814 INFO:teuthology.orchestra.run.smithi134.stdout:20 Mar 14:51:11 ntpd[9502]: ntpd: time slew -0.000905 s 2022-03-20T14:51:11.815 INFO:teuthology.orchestra.run.smithi134.stdout:ntpd: time slew -0.000905s 2022-03-20T14:51:11.815 INFO:teuthology.orchestra.run.smithi134.stderr:20 Mar 14:51:11 ntpd[9502]: can't open /var/log/ntpstats/loopstats.20220320: Permission denied 2022-03-20T14:51:12.093 INFO:teuthology.orchestra.run.smithi134.stdout: remote refid st t when poll reach delay offset jitter 2022-03-20T14:51:12.093 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================== 2022-03-20T14:51:12.093 INFO:teuthology.orchestra.run.smithi134.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:51:12.094 INFO:teuthology.orchestra.run.smithi134.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:51:12.094 INFO:teuthology.orchestra.run.smithi134.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:51:12.094 INFO:teuthology.orchestra.run.smithi134.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:51:12.095 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-20T14:51:12.113 INFO:tasks.kubeadm:Kubeadm start 2022-03-20T14:51:12.113 INFO:tasks.kubeadm:Config: {'version': '1.21'} 2022-03-20T14:51:12.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-20T14:51:12.142 INFO:teuthology.orchestra.run.smithi134.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-20T14:51:12.152 INFO:teuthology.orchestra.run.smithi134.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-20T14:51:12.161 INFO:teuthology.orchestra.run.smithi134.stdout:net.ipv4.ip_forward = 1 2022-03-20T14:51:12.198 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:51:12.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-20T14:51:12.255 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:51:12.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/docker/daemon.json 2022-03-20T14:51:12.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl restart docker || true 2022-03-20T14:51:13.229 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-20T14:51:13.236 INFO:tasks.kubeadm:172.21.15.134 is in 172.21.0.0/20 at pos 3973 2022-03-20T14:51:13.250 INFO:tasks.kubeadm:Pod subnet: 10.252.40.0/21 2022-03-20T14:51:13.251 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-20T14:51:13.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-20T14:51:13.317 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:51:13.318 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:51:13.318 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:51:13.430 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:51:13.438 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:51:13.439 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:51:13.459 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:51:14.905 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:51:15.076 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2022-03-20T14:51:15.077 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2022-03-20T14:51:15.170 INFO:teuthology.orchestra.run.smithi134.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T14:51:15.187 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:51:15.187 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:51:15.187 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:51:15.244 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:51:15.418 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2022-03-20T14:51:15.419 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2022-03-20T14:51:15.584 INFO:teuthology.orchestra.run.smithi134.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-20T14:51:15.585 INFO:teuthology.orchestra.run.smithi134.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-20T14:51:15.585 INFO:teuthology.orchestra.run.smithi134.stdout:The following NEW packages will be installed: 2022-03-20T14:51:15.585 INFO:teuthology.orchestra.run.smithi134.stdout: apt-transport-https 2022-03-20T14:51:15.643 INFO:teuthology.orchestra.run.smithi134.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T14:51:15.644 INFO:teuthology.orchestra.run.smithi134.stdout:Need to get 4,680 B of archives. 2022-03-20T14:51:15.644 INFO:teuthology.orchestra.run.smithi134.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-20T14:51:15.644 INFO:teuthology.orchestra.run.smithi134.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-20T14:51:15.910 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T14:51:15.910 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T14:51:15.911 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: falling back to frontend: Readline 2022-03-20T14:51:15.929 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T14:51:15.930 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T14:51:15.930 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: falling back to frontend: Teletype 2022-03-20T14:51:15.935 INFO:teuthology.orchestra.run.smithi134.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T14:51:15.984 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 4,680 B in 0s (92.2 kB/s) 2022-03-20T14:51:16.062 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package apt-transport-https. 2022-03-20T14:51:16.106 INFO:teuthology.orchestra.run.smithi134.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-20T14:51:16.109 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-20T14:51:16.144 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-20T14:51:16.388 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-20T14:51:16.881 INFO:teuthology.orchestra.run.smithi134.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-20T14:51:16.897 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:51:16.897 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:51:16.897 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:51:17.027 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:51:17.028 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:51:17.036 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:51:17.056 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:51:17.408 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-20T14:51:17.690 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-20T14:51:17.825 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 64.1 kB in 1s (82.0 kB/s) 2022-03-20T14:51:18.690 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:51:18.865 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2022-03-20T14:51:18.866 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2022-03-20T14:51:18.959 INFO:teuthology.orchestra.run.smithi134.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T14:51:18.972 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:51:18.972 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:51:18.972 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:51:19.027 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:51:19.187 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2022-03-20T14:51:19.187 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2022-03-20T14:51:19.317 INFO:teuthology.orchestra.run.smithi134.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-20T14:51:19.317 INFO:teuthology.orchestra.run.smithi134.stdout:bridge-utils set to manually installed. 2022-03-20T14:51:19.318 INFO:teuthology.orchestra.run.smithi134.stdout:The following additional packages will be installed: 2022-03-20T14:51:19.318 INFO:teuthology.orchestra.run.smithi134.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-20T14:51:19.319 INFO:teuthology.orchestra.run.smithi134.stdout:Suggested packages: 2022-03-20T14:51:19.319 INFO:teuthology.orchestra.run.smithi134.stdout: nftables 2022-03-20T14:51:19.371 INFO:teuthology.orchestra.run.smithi134.stdout:The following NEW packages will be installed: 2022-03-20T14:51:19.372 INFO:teuthology.orchestra.run.smithi134.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-20T14:51:19.442 INFO:teuthology.orchestra.run.smithi134.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T14:51:19.442 INFO:teuthology.orchestra.run.smithi134.stdout:Need to get 77.7 MB of archives. 2022-03-20T14:51:19.442 INFO:teuthology.orchestra.run.smithi134.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-20T14:51:19.443 INFO:teuthology.orchestra.run.smithi134.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-20T14:51:19.484 INFO:teuthology.orchestra.run.smithi134.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-20T14:51:19.510 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-20T14:51:19.621 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-20T14:51:20.212 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-20T14:51:21.648 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-20T14:51:22.186 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-20T14:51:22.426 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-20T14:51:22.865 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T14:51:22.866 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T14:51:22.866 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: falling back to frontend: Readline 2022-03-20T14:51:22.874 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T14:51:22.874 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T14:51:22.875 INFO:teuthology.orchestra.run.smithi134.stderr:debconf: falling back to frontend: Teletype 2022-03-20T14:51:22.880 INFO:teuthology.orchestra.run.smithi134.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T14:51:22.930 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 77.7 MB in 3s (23.8 MB/s) 2022-03-20T14:51:23.018 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package conntrack. 2022-03-20T14:51:23.065 INFO:teuthology.orchestra.run.smithi134.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-20T14:51:23.069 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-20T14:51:23.108 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-20T14:51:23.414 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package cri-tools. 2022-03-20T14:51:23.427 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-20T14:51:23.464 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-20T14:51:24.750 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ebtables. 2022-03-20T14:51:24.763 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-20T14:51:24.808 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-20T14:51:25.070 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-20T14:51:25.085 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-20T14:51:25.128 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-20T14:51:26.556 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package socat. 2022-03-20T14:51:26.571 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-20T14:51:26.605 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-20T14:51:26.970 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package kubelet. 2022-03-20T14:51:26.983 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-20T14:51:27.020 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-20T14:51:29.397 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package kubectl. 2022-03-20T14:51:29.410 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-20T14:51:29.447 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-20T14:51:30.600 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package kubeadm. 2022-03-20T14:51:30.612 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-20T14:51:30.641 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-20T14:51:31.755 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-20T14:51:31.880 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-20T14:51:31.991 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-20T14:51:32.789 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-20T14:51:32.917 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-20T14:51:33.036 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-20T14:51:33.163 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-20T14:51:33.389 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-20T14:51:33.877 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-20T14:51:34.266 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:51:35.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-20T14:51:48.380 INFO:teuthology.orchestra.run.smithi134.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-20T14:51:51.498 INFO:teuthology.orchestra.run.smithi134.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-20T14:51:53.671 INFO:teuthology.orchestra.run.smithi134.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-20T14:51:57.020 INFO:teuthology.orchestra.run.smithi134.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-20T14:51:58.311 INFO:teuthology.orchestra.run.smithi134.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-20T14:52:04.260 INFO:teuthology.orchestra.run.smithi134.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-20T14:52:06.968 INFO:teuthology.orchestra.run.smithi134.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-20T14:52:06.973 INFO:tasks.kubeadm:Token: abcdef.9pam48ofk56shw2u 2022-03-20T14:52:06.973 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): '172.21.15.134'} 2022-03-20T14:52:06.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo kubeadm init --node-name smithi134 --token abcdef.9pam48ofk56shw2u --pod-network-cidr 10.252.40.0/21 2022-03-20T14:52:07.183 INFO:teuthology.orchestra.run.smithi134.stdout:[init] Using Kubernetes version: v1.23.5 2022-03-20T14:52:07.183 INFO:teuthology.orchestra.run.smithi134.stdout:[preflight] Running pre-flight checks 2022-03-20T14:52:07.568 INFO:teuthology.orchestra.run.smithi134.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-20T14:52:07.569 INFO:teuthology.orchestra.run.smithi134.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-20T14:52:07.569 INFO:teuthology.orchestra.run.smithi134.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-20T14:52:07.890 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-20T14:52:08.005 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "ca" certificate and key 2022-03-20T14:52:08.169 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "apiserver" certificate and key 2022-03-20T14:52:08.170 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi134] and IPs [10.96.0.1 172.21.15.134] 2022-03-20T14:52:08.283 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-20T14:52:08.566 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-20T14:52:08.669 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-20T14:52:08.797 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-20T14:52:08.948 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-20T14:52:08.948 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi134] and IPs [172.21.15.134 127.0.0.1 ::1] 2022-03-20T14:52:09.086 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-20T14:52:09.086 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi134] and IPs [172.21.15.134 127.0.0.1 ::1] 2022-03-20T14:52:09.275 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-20T14:52:09.351 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-20T14:52:09.551 INFO:teuthology.orchestra.run.smithi134.stdout:[certs] Generating "sa" key and public key 2022-03-20T14:52:09.551 INFO:teuthology.orchestra.run.smithi134.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-20T14:52:09.722 INFO:teuthology.orchestra.run.smithi134.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-20T14:52:09.883 INFO:teuthology.orchestra.run.smithi134.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-20T14:52:09.930 INFO:teuthology.orchestra.run.smithi134.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-20T14:52:10.089 INFO:teuthology.orchestra.run.smithi134.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-20T14:52:10.101 INFO:teuthology.orchestra.run.smithi134.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-20T14:52:10.104 INFO:teuthology.orchestra.run.smithi134.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-20T14:52:10.104 INFO:teuthology.orchestra.run.smithi134.stdout:[kubelet-start] Starting the kubelet 2022-03-20T14:52:10.327 INFO:teuthology.orchestra.run.smithi134.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-20T14:52:10.328 INFO:teuthology.orchestra.run.smithi134.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-20T14:52:10.330 INFO:teuthology.orchestra.run.smithi134.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-20T14:52:10.332 INFO:teuthology.orchestra.run.smithi134.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-20T14:52:10.333 INFO:teuthology.orchestra.run.smithi134.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-20T14:52:10.335 INFO:teuthology.orchestra.run.smithi134.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-20T14:52:26.845 INFO:teuthology.orchestra.run.smithi134.stdout:[apiclient] All control plane components are healthy after 16.509867 seconds 2022-03-20T14:52:26.845 INFO:teuthology.orchestra.run.smithi134.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-20T14:52:26.920 INFO:teuthology.orchestra.run.smithi134.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-20T14:52:26.920 INFO:teuthology.orchestra.run.smithi134.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-20T14:52:27.474 INFO:teuthology.orchestra.run.smithi134.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-20T14:52:27.475 INFO:teuthology.orchestra.run.smithi134.stdout:[mark-control-plane] Marking the node smithi134 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-20T14:52:27.475 INFO:teuthology.orchestra.run.smithi134.stdout:[mark-control-plane] Marking the node smithi134 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-20T14:52:28.003 INFO:teuthology.orchestra.run.smithi134.stdout:[bootstrap-token] Using token: abcdef.9pam48ofk56shw2u 2022-03-20T14:52:28.004 INFO:teuthology.orchestra.run.smithi134.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-20T14:52:28.092 INFO:teuthology.orchestra.run.smithi134.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-20T14:52:28.130 INFO:teuthology.orchestra.run.smithi134.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-20T14:52:28.156 INFO:teuthology.orchestra.run.smithi134.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-20T14:52:28.161 INFO:teuthology.orchestra.run.smithi134.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-20T14:52:28.169 INFO:teuthology.orchestra.run.smithi134.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-20T14:52:28.289 INFO:teuthology.orchestra.run.smithi134.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-20T14:52:28.711 INFO:teuthology.orchestra.run.smithi134.stdout:[addons] Applied essential addon: CoreDNS 2022-03-20T14:52:28.795 INFO:teuthology.orchestra.run.smithi134.stdout:[addons] Applied essential addon: kube-proxy 2022-03-20T14:52:28.796 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:52:28.797 INFO:teuthology.orchestra.run.smithi134.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-20T14:52:28.797 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:52:28.797 INFO:teuthology.orchestra.run.smithi134.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-20T14:52:28.798 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:52:28.798 INFO:teuthology.orchestra.run.smithi134.stdout: mkdir -p $HOME/.kube 2022-03-20T14:52:28.798 INFO:teuthology.orchestra.run.smithi134.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-20T14:52:28.799 INFO:teuthology.orchestra.run.smithi134.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-20T14:52:28.799 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:52:28.800 INFO:teuthology.orchestra.run.smithi134.stdout:Alternatively, if you are the root user, you can run: 2022-03-20T14:52:28.800 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:52:28.800 INFO:teuthology.orchestra.run.smithi134.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-20T14:52:28.801 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:52:28.801 INFO:teuthology.orchestra.run.smithi134.stdout:You should now deploy a pod network to the cluster. 2022-03-20T14:52:28.801 INFO:teuthology.orchestra.run.smithi134.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-20T14:52:28.802 INFO:teuthology.orchestra.run.smithi134.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-20T14:52:28.802 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:52:28.802 INFO:teuthology.orchestra.run.smithi134.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-20T14:52:28.803 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:52:28.803 INFO:teuthology.orchestra.run.smithi134.stdout:kubeadm join 172.21.15.134:6443 --token abcdef.9pam48ofk56shw2u \ 2022-03-20T14:52:28.803 INFO:teuthology.orchestra.run.smithi134.stdout: --discovery-token-ca-cert-hash sha256:f62edff11cd67dc0d692c3b9069b9134e9d1e1d9cbf856e16d0cd6cf4fed4714 2022-03-20T14:52:28.805 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:52:28.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-20T14:52:28.818 INFO:tasks.kubeadm:Setting up kubectl 2022-03-20T14:52:28.819 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-20T14:52:28.830 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:52:28.830 DEBUG:teuthology.orchestra.run.smithi134:> dd of=.kube/config 2022-03-20T14:52:28.877 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:52:28.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/root/.kube/config 2022-03-20T14:52:28.889 DEBUG:teuthology.orchestra.run.smithi134:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-20T14:52:31.492 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-20T14:52:31.591 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-20T14:52:31.640 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-20T14:52:31.753 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-20T14:52:31.809 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-20T14:52:31.914 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-20T14:52:31.953 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-20T14:52:32.000 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-20T14:52:32.114 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-20T14:52:32.158 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-20T14:52:32.258 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-20T14:52:32.292 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-20T14:52:32.342 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-20T14:52:32.422 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-20T14:52:32.469 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-20T14:52:32.523 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-20T14:52:32.603 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-20T14:52:32.652 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-20T14:52:32.761 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-20T14:52:32.827 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-20T14:52:32.935 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-20T14:52:32.968 INFO:teuthology.orchestra.run.smithi134.stdout:namespace/tigera-operator created 2022-03-20T14:52:32.992 INFO:teuthology.orchestra.run.smithi134.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-20T14:52:32.997 INFO:teuthology.orchestra.run.smithi134.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-20T14:52:33.005 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/tigera-operator created 2022-03-20T14:52:33.012 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-20T14:52:33.077 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-20T14:52:33.119 INFO:teuthology.orchestra.run.smithi134.stdout:deployment.apps/tigera-operator created 2022-03-20T14:52:33.124 DEBUG:teuthology.orchestra.run.smithi134:> kubectl create -f - 2022-03-20T14:52:35.946 INFO:teuthology.orchestra.run.smithi134.stdout:installation.operator.tigera.io/default created 2022-03-20T14:52:35.952 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-20T14:52:35.952 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:52:35.953 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:35.958 INFO:tasks.kubeadm:Scratch devices on smithi134: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T14:52:35.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-20T14:52:35.986 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records in 2022-03-20T14:52:35.987 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2022-03-20T14:52:35.987 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163347 s, 642 MB/s 2022-03-20T14:52:35.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-20T14:52:36.014 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records in 2022-03-20T14:52:36.014 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2022-03-20T14:52:36.014 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0170432 s, 615 MB/s 2022-03-20T14:52:36.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-20T14:52:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records in 2022-03-20T14:52:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2022-03-20T14:52:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0157738 s, 665 MB/s 2022-03-20T14:52:36.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-20T14:52:36.117 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records in 2022-03-20T14:52:36.117 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2022-03-20T14:52:36.118 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148107 s, 708 MB/s 2022-03-20T14:52:36.125 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-20T14:52:36.125 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi134-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi134 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi134-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi134 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi134-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi134 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi134-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi134 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-20T14:52:36.126 DEBUG:teuthology.orchestra.run.smithi134:> kubectl create -f - 2022-03-20T14:52:37.010 INFO:teuthology.orchestra.run.smithi134.stdout:persistentvolume/smithi134-lv-1 created 2022-03-20T14:52:37.018 INFO:teuthology.orchestra.run.smithi134.stdout:persistentvolume/smithi134-lv-2 created 2022-03-20T14:52:37.026 INFO:teuthology.orchestra.run.smithi134.stdout:persistentvolume/smithi134-lv-3 created 2022-03-20T14:52:37.035 INFO:teuthology.orchestra.run.smithi134.stdout:persistentvolume/smithi134-lv-4 created 2022-03-20T14:52:37.059 INFO:teuthology.orchestra.run.smithi134.stdout:storageclass.storage.k8s.io/scratch created 2022-03-20T14:52:37.065 DEBUG:teuthology.orchestra.run.smithi134:> kubectl taint node smithi134 node-role.kubernetes.io/master- || true 2022-03-20T14:52:37.131 INFO:teuthology.orchestra.run.smithi134.stdout:node/smithi134 untainted 2022-03-20T14:52:37.134 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-20T14:52:37.134 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-20T14:52:37.148 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-20T14:52:37.149 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:52:37.149 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:37.156 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T14:52:37.157 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-20T14:52:37.162 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-20T14:52:37.163 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:37.163 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-20T14:52:37.163 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:37.163 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2022-03-20 14:52:35.996560289 +0000 2022-03-20T14:52:37.164 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2022-03-20 14:52:35.996560289 +0000 2022-03-20T14:52:37.164 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2022-03-20 14:52:35.996560289 +0000 2022-03-20T14:52:37.164 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2022-03-20T14:52:37.172 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2022-03-20T14:52:37.172 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2022-03-20T14:52:37.172 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000573693 s, 892 kB/s 2022-03-20T14:52:37.176 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-20T14:52:37.183 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-20T14:52:37.184 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:37.184 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-20T14:52:37.184 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:37.184 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2022-03-20 14:52:36.016560289 +0000 2022-03-20T14:52:37.185 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2022-03-20 14:52:36.016560289 +0000 2022-03-20T14:52:37.185 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2022-03-20 14:52:36.016560289 +0000 2022-03-20T14:52:37.185 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2022-03-20T14:52:37.191 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2022-03-20T14:52:37.191 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2022-03-20T14:52:37.192 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000491407 s, 1.0 MB/s 2022-03-20T14:52:37.197 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-20T14:52:37.203 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-20T14:52:37.204 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:37.204 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-20T14:52:37.204 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:37.204 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2022-03-20 14:52:36.092560286 +0000 2022-03-20T14:52:37.205 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2022-03-20 14:52:36.092560286 +0000 2022-03-20T14:52:37.205 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2022-03-20 14:52:36.092560286 +0000 2022-03-20T14:52:37.205 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2022-03-20T14:52:37.210 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2022-03-20T14:52:37.211 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2022-03-20T14:52:37.211 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000562425 s, 910 kB/s 2022-03-20T14:52:37.215 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-20T14:52:37.265 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-20T14:52:37.265 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:37.266 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-20T14:52:37.266 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:37.266 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2022-03-20 14:52:36.120560286 +0000 2022-03-20T14:52:37.266 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2022-03-20 14:52:36.120560286 +0000 2022-03-20T14:52:37.267 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2022-03-20 14:52:36.120560286 +0000 2022-03-20T14:52:37.267 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2022-03-20T14:52:37.274 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2022-03-20T14:52:37.274 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2022-03-20T14:52:37.274 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000504529 s, 1.0 MB/s 2022-03-20T14:52:37.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-20T14:52:37.377 INFO:teuthology.orchestra.run.smithi134.stdout:loop 2022-03-20T14:52:37.379 INFO:tasks.nvme_loop:Connecting nvme_loop smithi134:/dev/vg_nvme/lv_1... 2022-03-20T14:52:37.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-20T14:52:37.401 INFO:teuthology.orchestra.run.smithi134.stdout:1 2022-03-20T14:52:37.425 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/vg_nvme/lv_11 2022-03-20T14:52:37.448 INFO:tasks.nvme_loop:Connecting nvme_loop smithi134:/dev/vg_nvme/lv_2... 2022-03-20T14:52:37.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-20T14:52:37.511 INFO:teuthology.orchestra.run.smithi134.stdout:1 2022-03-20T14:52:37.536 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/vg_nvme/lv_21 2022-03-20T14:52:37.559 INFO:tasks.nvme_loop:Connecting nvme_loop smithi134:/dev/vg_nvme/lv_3... 2022-03-20T14:52:37.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-20T14:52:37.579 INFO:teuthology.orchestra.run.smithi134.stdout:1 2022-03-20T14:52:37.599 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/vg_nvme/lv_31 2022-03-20T14:52:37.621 INFO:tasks.nvme_loop:Connecting nvme_loop smithi134:/dev/vg_nvme/lv_4... 2022-03-20T14:52:37.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-20T14:52:37.683 INFO:teuthology.orchestra.run.smithi134.stdout:1 2022-03-20T14:52:37.706 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/vg_nvme/lv_41 2022-03-20T14:52:37.729 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:52:37.730 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:37.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo nvme list 2022-03-20T14:52:37.749 INFO:teuthology.orchestra.run.smithi134.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-20T14:52:37.750 INFO:teuthology.orchestra.run.smithi134.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-20T14:52:37.750 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/nvme0n1 CVFT62420002400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-20T14:52:37.750 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/nvme1n1 2ba2bceaac1f829e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:37.751 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/nvme2n1 01844bf0c59e6fed Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:37.751 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/nvme3n1 738ac0c4afe71690 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:37.752 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-20T14:52:38.753 DEBUG:teuthology.orchestra.run.smithi134:> sudo nvme list 2022-03-20T14:52:38.769 INFO:teuthology.orchestra.run.smithi134.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-20T14:52:38.769 INFO:teuthology.orchestra.run.smithi134.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-20T14:52:38.770 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/nvme0n1 CVFT62420002400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-20T14:52:38.770 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/nvme1n1 2ba2bceaac1f829e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:38.770 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/nvme2n1 01844bf0c59e6fed Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:38.770 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/nvme3n1 738ac0c4afe71690 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:38.771 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/nvme4n1 c733e733bf4c7d9e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:38.772 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-20T14:52:38.772 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:52:38.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/scratch_devs 2022-03-20T14:52:38.786 INFO:teuthology.run_tasks:Running task rook... 2022-03-20T14:52:38.858 INFO:tasks.rook:Rook start 2022-03-20T14:52:38.859 INFO:tasks.rook:Config: {'conf': {'global': {'osd crush chooseleaf type': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e', 'spec': {'mon': {'allowMultiplePerNode': False}, 'network': {'provider': 'host'}}, 'rook_branch': 'v1.7.2', 'rook_image': 'rook/ceph:v1.7.2'} 2022-03-20T14:52:38.859 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-20T14:52:38.859 INFO:tasks.rook:Cloning https://github.com/rook/rook branch v1.7.2 2022-03-20T14:52:38.860 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf rook && git clone --single-branch --branch v1.7.2 https://github.com/rook/rook rook 2022-03-20T14:52:38.868 INFO:teuthology.orchestra.run.smithi134.stderr:Cloning into 'rook'... 2022-03-20T14:52:41.102 INFO:teuthology.orchestra.run.smithi134.stderr:Note: switching to '1784489964d2e417b6050c7edc8f626b5cca49f5'. 2022-03-20T14:52:41.103 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:52:41.103 INFO:teuthology.orchestra.run.smithi134.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-03-20T14:52:41.103 INFO:teuthology.orchestra.run.smithi134.stderr:changes and commit them, and you can discard any commits you make in this 2022-03-20T14:52:41.103 INFO:teuthology.orchestra.run.smithi134.stderr:state without impacting any branches by switching back to a branch. 2022-03-20T14:52:41.104 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:52:41.104 INFO:teuthology.orchestra.run.smithi134.stderr:If you want to create a new branch to retain commits you create, you may 2022-03-20T14:52:41.104 INFO:teuthology.orchestra.run.smithi134.stderr:do so (now or later) by using -c with the switch command. Example: 2022-03-20T14:52:41.104 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:52:41.105 INFO:teuthology.orchestra.run.smithi134.stderr: git switch -c 2022-03-20T14:52:41.105 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:52:41.105 INFO:teuthology.orchestra.run.smithi134.stderr:Or undo this operation with: 2022-03-20T14:52:41.105 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:52:41.105 INFO:teuthology.orchestra.run.smithi134.stderr: git switch - 2022-03-20T14:52:41.106 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:52:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-03-20T14:52:41.106 INFO:teuthology.orchestra.run.smithi134.stderr: 2022-03-20T14:52:41.209 INFO:tasks.rook:/home/teuthworker 2022-03-20T14:52:41.210 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/sbin:/usr/sbin ip addr show 2022-03-20T14:52:41.261 INFO:teuthology.orchestra.run.smithi134.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-20T14:52:41.261 INFO:teuthology.orchestra.run.smithi134.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-20T14:52:41.261 INFO:teuthology.orchestra.run.smithi134.stdout: inet 127.0.0.1/8 scope host lo 2022-03-20T14:52:41.261 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2022-03-20T14:52:41.262 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 ::1/128 scope host 2022-03-20T14:52:41.262 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2022-03-20T14:52:41.262 INFO:teuthology.orchestra.run.smithi134.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-20T14:52:41.262 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e6 brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:41.263 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f0 2022-03-20T14:52:41.263 INFO:teuthology.orchestra.run.smithi134.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-20T14:52:41.263 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e7 brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:41.263 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f1 2022-03-20T14:52:41.264 INFO:teuthology.orchestra.run.smithi134.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-20T14:52:41.264 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e8 brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:41.264 INFO:teuthology.orchestra.run.smithi134.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-20T14:52:41.265 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e9 brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:41.265 INFO:teuthology.orchestra.run.smithi134.stdout: inet 172.21.15.134/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-20T14:52:41.265 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft 42880sec preferred_lft 42880sec 2022-03-20T14:52:41.265 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 fe80::ec4:7aff:fe88:6de9/64 scope link 2022-03-20T14:52:41.266 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2022-03-20T14:52:41.266 INFO:teuthology.orchestra.run.smithi134.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-20T14:52:41.266 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 02:42:05:5e:3a:bb brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:41.266 INFO:teuthology.orchestra.run.smithi134.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-20T14:52:41.266 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2022-03-20T14:52:41.268 DEBUG:teuthology.orchestra.run.smithi134:> which systemctl 2022-03-20T14:52:41.274 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/systemctl 2022-03-20T14:52:41.275 DEBUG:teuthology.orchestra.run.smithi134:> true 2022-03-20T14:52:41.350 DEBUG:teuthology.orchestra.run.smithi134:> sudo lsb_release -is 2022-03-20T14:52:41.395 INFO:teuthology.orchestra.run.smithi134.stdout:Ubuntu 2022-03-20T14:52:41.400 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-20T14:52:41.400 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-20T14:52:41.417 DEBUG:teuthology.orchestra.remote:smithi134:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-20T14:52:41.430 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:52:41.430 DEBUG:teuthology.orchestra.run.smithi134:> dd if=rook/cluster/examples/kubernetes/ceph/operator.yaml of=/dev/stdout 2022-03-20T14:52:41.438 INFO:tasks.rook:Patching operator to use image rook/ceph:v1.7.2 2022-03-20T14:52:41.479 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:52:41.480 DEBUG:teuthology.orchestra.run.smithi134:> dd of=operator.yaml 2022-03-20T14:52:41.492 INFO:tasks.rook:Deploying operator 2022-03-20T14:52:41.509 DEBUG:teuthology.orchestra.remote:smithi134:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-20T14:52:41.542 DEBUG:teuthology.orchestra.remote:smithi134:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-20T14:52:41.553 DEBUG:teuthology.orchestra.run.smithi134:> kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml 2022-03-20T14:52:41.866 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-20T14:52:41.959 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-20T14:52:42.073 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-20T14:52:42.147 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-20T14:52:42.196 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-20T14:52:42.218 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-20T14:52:42.239 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-20T14:52:42.344 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-20T14:52:42.368 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-20T14:52:42.389 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-20T14:52:42.418 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-20T14:52:42.515 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-20T14:52:42.545 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-20T14:52:42.591 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-20T14:52:42.739 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created 2022-03-20T14:52:42.835 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created 2022-03-20T14:52:43.188 INFO:teuthology.orchestra.run.smithi134.stdout:customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created 2022-03-20T14:52:43.574 INFO:teuthology.orchestra.run.smithi134.stdout:namespace/rook-ceph created 2022-03-20T14:52:43.623 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-20T14:52:43.726 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-ceph-admission-controller created 2022-03-20T14:52:43.796 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role created 2022-03-20T14:52:43.956 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding created 2022-03-20T14:52:44.041 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-20T14:52:44.133 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T14:52:44.137 INFO:teuthology.orchestra.run.smithi134.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T14:52:44.168 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-20T14:52:44.184 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-20T14:52:44.225 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-20T14:52:44.308 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-ceph-system created 2022-03-20T14:52:44.311 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T14:52:44.351 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T14:52:44.359 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-20T14:52:44.401 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-ceph-osd created 2022-03-20T14:52:44.456 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-ceph-mgr created 2022-03-20T14:52:44.458 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-20T14:52:44.493 INFO:teuthology.orchestra.run.smithi134.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T14:52:44.543 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T14:52:44.635 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-20T14:52:44.638 INFO:teuthology.orchestra.run.smithi134.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-20T14:52:44.683 INFO:teuthology.orchestra.run.smithi134.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-20T14:52:44.692 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-20T14:52:44.826 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T14:52:44.830 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-20T14:52:44.855 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-20T14:52:44.867 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-20T14:52:44.876 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T14:52:44.884 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-20T14:52:44.999 INFO:teuthology.orchestra.run.smithi134.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-20T14:52:45.000 INFO:teuthology.orchestra.run.smithi134.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-20T14:52:45.003 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-20T14:52:45.016 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-20T14:52:45.045 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-20T14:52:45.051 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-20T14:52:45.072 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-20T14:52:45.180 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-20T14:52:45.183 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-20T14:52:45.209 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-20T14:52:45.217 INFO:teuthology.orchestra.run.smithi134.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-20T14:52:45.275 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-20T14:52:45.626 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-20T14:52:45.817 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-20T14:52:45.821 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-20T14:52:45.850 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-20T14:52:46.031 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-20T14:52:46.034 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-20T14:52:46.118 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-20T14:52:46.201 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-20T14:52:46.206 INFO:teuthology.orchestra.run.smithi134.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-20T14:52:46.237 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-20T14:52:46.278 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-20T14:52:46.284 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-20T14:52:46.364 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-20T14:52:46.367 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-20T14:52:46.395 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-20T14:52:46.409 INFO:teuthology.orchestra.run.smithi134.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-20T14:52:46.428 INFO:teuthology.orchestra.run.smithi134.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-20T14:52:46.442 INFO:teuthology.orchestra.run.smithi134.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-20T14:52:46.539 INFO:teuthology.orchestra.run.smithi134.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-20T14:52:46.545 INFO:teuthology.orchestra.run.smithi134.stdout:configmap/rook-ceph-operator-config created 2022-03-20T14:52:46.575 INFO:teuthology.orchestra.run.smithi134.stdout:deployment.apps/rook-ceph-operator created 2022-03-20T14:52:46.580 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:52:46.718 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:52:46.718 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 0/1 Pending 0 0s 2022-03-20T14:52:56.723 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:52:57.207 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:52:57.208 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 0/1 Pending 0 11s 2022-03-20T14:53:07.211 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:53:07.628 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:07.628 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 0/1 Pending 0 21s 2022-03-20T14:53:17.631 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:53:17.876 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:17.876 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 0/1 Pending 0 31s 2022-03-20T14:53:27.879 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:53:28.079 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:28.080 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 0/1 Pending 0 42s 2022-03-20T14:53:38.083 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:53:38.152 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:38.153 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 0/1 ContainerCreating 0 52s 2022-03-20T14:53:48.154 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:53:48.224 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:48.224 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 0/1 ContainerCreating 0 62s 2022-03-20T14:53:58.228 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:53:58.614 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:58.614 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 0/1 ContainerCreating 0 72s 2022-03-20T14:54:08.616 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:54:08.686 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:08.686 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 0/1 ContainerCreating 0 82s 2022-03-20T14:54:18.694 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:54:18.768 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:18.769 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-operator-6b496c8959-nvg8b 1/1 Running 0 92s 2022-03-20T14:54:18.770 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph logs -f rook-ceph-operator-6b496c8959-nvg8b 2022-03-20T14:54:18.777 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:54:18.777 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:54:18.825 INFO:tasks.rook: override: [global] osd crush chooseleaf type = 0 2022-03-20T14:54:18.825 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-20T14:54:18.826 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-20T14:54:18.826 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-20T14:54:18.826 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-20T14:54:18.827 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-20T14:54:18.827 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-20T14:54:18.827 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-20T14:54:18.828 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-20T14:54:18.830 DEBUG:teuthology.orchestra.run.smithi134:> kubectl create -f - 2022-03-20T14:54:18.861 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.341541 I | rookcmd: starting Rook v1.7.2 with arguments '/usr/local/bin/rook ceph operator' 2022-03-20T14:54:18.862 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.341646 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule= 2022-03-20T14:54:18.862 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.341650 I | cephcmd: starting Rook-Ceph operator 2022-03-20T14:54:18.862 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.506669 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)" 2022-03-20T14:54:18.862 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.511461 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T14:54:18.863 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.515521 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T14:54:18.863 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.518714 I | operator: looking for secret "rook-ceph-admission-controller" 2022-03-20T14:54:18.863 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.519927 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T14:54:18.863 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.521802 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-20T14:54:18.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.521814 I | operator: watching all namespaces for ceph cluster CRs 2022-03-20T14:54:18.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.521902 I | operator: setting up the controller-runtime manager 2022-03-20T14:54:18.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.523332 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations 2022-03-20T14:54:18.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.525396 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T14:54:18.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930155 I | ceph-cluster-controller: successfully started 2022-03-20T14:54:18.865 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930260 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T14:54:18.865 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930281 I | ceph-crashcollector-controller: successfully started 2022-03-20T14:54:18.865 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930351 I | ceph-block-pool-controller: successfully started 2022-03-20T14:54:18.865 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930434 I | ceph-object-store-user-controller: successfully started 2022-03-20T14:54:18.866 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930514 I | ceph-object-realm-controller: successfully started 2022-03-20T14:54:18.866 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930582 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T14:54:18.866 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930662 I | ceph-object-zone-controller: successfully started 2022-03-20T14:54:18.866 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930825 I | ceph-object-controller: successfully started 2022-03-20T14:54:18.867 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.930933 I | ceph-file-controller: successfully started 2022-03-20T14:54:18.867 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.931023 I | ceph-nfs-controller: successfully started 2022-03-20T14:54:18.867 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.931113 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T14:54:18.867 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.931235 I | ceph-client-controller: successfully started 2022-03-20T14:54:18.867 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.931316 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T14:54:18.868 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:11.933045 I | operator: starting the controller-runtime manager 2022-03-20T14:54:19.762 INFO:teuthology.orchestra.run.smithi134.stdout:configmap/rook-config-override created 2022-03-20T14:54:19.767 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: false count: 1 network: provider: host skipUpgradeChecks: true 2022-03-20T14:54:19.767 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:54:19.768 DEBUG:teuthology.orchestra.run.smithi134:> dd of=cluster.yaml 2022-03-20T14:54:19.819 DEBUG:teuthology.orchestra.run.smithi134:> kubectl create -f cluster.yaml 2022-03-20T14:54:22.150 INFO:teuthology.orchestra.run.smithi134.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-20T14:54:22.155 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.156855 I | clusterdisruption-controller: create event from ceph cluster CR 2022-03-20T14:54:22.156 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.157028 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-20T14:54:22.162 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.164988 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:54:22.211 DEBUG:teuthology.orchestra.remote:smithi134:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-20T14:54:22.216 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.218676 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-03-20T14:54:22.226 DEBUG:teuthology.orchestra.run.smithi134:> kubectl create -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-20T14:54:22.232 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.234702 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-20T14:54:22.241 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.244305 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster 2022-03-20T14:54:22.242 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.244393 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-20T14:54:22.245 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.246738 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-20T14:54:22.250 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.248795 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-20T14:54:22.251 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.251050 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-20T14:54:22.251 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.252971 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-20T14:54:22.259 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.261933 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T14:54:22.403 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.405251 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.4.0" (default) 2022-03-20T14:54:22.523 INFO:teuthology.orchestra.run.smithi134.stdout:deployment.apps/rook-ceph-tools created 2022-03-20T14:54:22.527 INFO:tasks.rook:Waiting for tools container to start 2022-03-20T14:54:22.527 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:22.727 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:22.728 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 Pending 0 0s 2022-03-20T14:54:22.788 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:22.790592 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0" (default) 2022-03-20T14:54:23.168 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:23.169946 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2" (default) 2022-03-20T14:54:23.366 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:23.368427 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.2.1" (default) 2022-03-20T14:54:23.690 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:23.693082 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1" (default) 2022-03-20T14:54:23.767 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:23.769231 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-20T14:54:23.966 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:23.968972 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default) 2022-03-20T14:54:24.185 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:24.187915 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-20T14:54:24.366 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:24.368487 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-20T14:54:24.366 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:24.368520 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.4.0" 2022-03-20T14:54:24.568 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:54:24.570157 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:54:27.730 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:27.864 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:27.865 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 5s 2022-03-20T14:54:32.867 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:32.948 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:32.948 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 10s 2022-03-20T14:54:37.951 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:38.014 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:38.015 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 16s 2022-03-20T14:54:43.018 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:43.085 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:43.086 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 21s 2022-03-20T14:54:48.088 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:48.146 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:48.146 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 26s 2022-03-20T14:54:53.150 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:53.211 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:53.212 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 31s 2022-03-20T14:54:58.214 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:58.268 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:58.268 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 36s 2022-03-20T14:55:01.520 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:01.520246 I | ceph-csi: Detected ceph CSI image version: "v3.4.0" 2022-03-20T14:55:01.551 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:01.529143 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-20T14:55:01.556 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:01.551558 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T14:55:01.564 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:01.563977 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T14:55:01.653 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:01.652473 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T14:55:01.703 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:01.702988 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T14:55:02.247 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.246916 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-20T14:55:02.270 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.269828 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-20T14:55:02.344 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.344046 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-20T14:55:02.439 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.438644 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-20T14:55:02.442 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.440427 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T14:55:02.442 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.440450 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T14:55:02.442 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.440464 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11187 quincy", pursuing 2022-03-20T14:55:02.471 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.470598 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-20T14:55:02.601 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.601180 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T14:55:02.778 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.777633 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-20T14:55:02.963 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:02.962537 I | op-mon: start running mons 2022-03-20T14:55:03.093 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:03.092703 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T14:55:03.271 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:03.337 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:03.337 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 41s 2022-03-20T14:55:03.348 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:03.348983 I | op-mon: creating mon secrets for a new cluster 2022-03-20T14:55:03.927 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:03.926647 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T14:55:03.927 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:03.926678 I | ceph-csi: Kubernetes version is 1.23 2022-03-20T14:55:03.929 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:03.929244 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0" (default) 2022-03-20T14:55:04.300 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:04.300511 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-20T14:55:04.493 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:04.493504 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-20T14:55:04.895 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:04.895113 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-20T14:55:05.093 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:05.093177 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:55:05.514 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:05.514140 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:55:05.894 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:05.894261 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-20T14:55:06.297 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:06.296438 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T14:55:06.493 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:06.493091 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:55:06.493 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:06.493312 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:55:06.726 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:06.725389 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-20T14:55:07.093 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:07.093312 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T14:55:07.494 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:07.494012 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-20T14:55:07.893 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:07.893507 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:55:08.340 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:08.394 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:08.394108 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:55:08.480 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:08.481 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 46s 2022-03-20T14:55:08.694 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:08.693875 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-20T14:55:08.708 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:08.708790 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-20T14:55:08.928 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:08.927324 I | op-mon: targeting the mon count 1 2022-03-20T14:55:09.073 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:09.073378 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-20T14:55:09.501 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:09.500741 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-20T14:55:09.736 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:09.736161 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T14:55:09.893 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:09.893654 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-20T14:55:10.343 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:10.343637 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:55:10.346 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:10.346172 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:55:10.701 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:10.700890 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-20T14:55:11.252 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:11.252804 I | ceph-csi: successfully started CSI CephFS driver 2022-03-20T14:55:11.891 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:11.890042 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T14:55:12.475 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:12.475358 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-20T14:55:12.514 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:12.514221 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-03-20T14:55:13.283 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:13.283045 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-20T14:55:13.485 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:13.545 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:13.546 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 51s 2022-03-20T14:55:14.383 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:14.383517 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi134 2022-03-20T14:55:14.384 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:14.383544 I | op-mon: assignmon: mon a assigned to node smithi134 2022-03-20T14:55:14.390 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:14.387446 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-20T14:55:14.394 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:14.391972 I | op-mon: creating mon a 2022-03-20T14:55:14.394 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:14.392003 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-20T14:55:14.609 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:14.609131 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:-1] 2022-03-20T14:55:14.609 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:14.609277 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-20T14:55:14.610 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:14.609315 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-20T14:55:15.025 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:15.024829 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:55:15.179 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:15.105992 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:55:15.898 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:15.898415 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-20T14:55:16.166 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:16.166276 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-20T14:55:17.218 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:17.216818 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T14:55:17.218 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:17.216876 I | op-mon: waiting for mon quorum with [a] 2022-03-20T14:55:17.224 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:17.222599 I | op-mon: mon a is not yet running 2022-03-20T14:55:17.225 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:17.222629 I | op-mon: mons running: [] 2022-03-20T14:55:18.549 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:18.681 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:18.681 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 56s 2022-03-20T14:55:23.683 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:23.849 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:23.849 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 61s 2022-03-20T14:55:28.852 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:29.034 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:29.035 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 67s 2022-03-20T14:55:34.039 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:34.181 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:34.181 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 0/1 ContainerCreating 0 72s 2022-03-20T14:55:37.333 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:37.333707 I | op-mon: mon a is not yet running 2022-03-20T14:55:37.333 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:37.333732 I | op-mon: mons running: [] 2022-03-20T14:55:38.971 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:38.971282 I | op-mon: Monitors in quorum: [a] 2022-03-20T14:55:38.971 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:38.971312 I | op-mon: mons created: 1 2022-03-20T14:55:39.180 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:39.181347 I | op-mon: waiting for mon quorum with [a] 2022-03-20T14:55:39.184 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:39.370 INFO:teuthology.orchestra.run.smithi134.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:39.371 INFO:teuthology.orchestra.run.smithi134.stdout:rook-ceph-tools-7677f7f4b9-xpmf2 1/1 Running 0 77s 2022-03-20T14:55:39.371 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:39.371803 I | op-mon: mons running: [a] 2022-03-20T14:55:39.373 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-20T14:55:39.373 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch status -f json 2022-03-20T14:55:39.732 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:39.733493 I | op-mon: Monitors in quorum: [a] 2022-03-20T14:55:39.733 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:39.733531 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:55:40.198 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:55:40.199 INFO:teuthology.orchestra.run.smithi134.stderr:no valid command found; 10 closest matches: 2022-03-20T14:55:40.199 INFO:teuthology.orchestra.run.smithi134.stderr:pg stat 2022-03-20T14:55:40.199 INFO:teuthology.orchestra.run.smithi134.stderr:pg getmap 2022-03-20T14:55:40.199 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:55:40.200 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:55:40.200 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_pools_json 2022-03-20T14:55:40.200 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-pool [...] 2022-03-20T14:55:40.200 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-primary [] [...] 2022-03-20T14:55:40.201 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-osd [] [...] 2022-03-20T14:55:40.201 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls [] [...] 2022-03-20T14:55:40.201 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:55:40.201 INFO:teuthology.orchestra.run.smithi134.stderr:Error EINVAL: invalid command 2022-03-20T14:55:40.203 INFO:teuthology.orchestra.run.smithi134.stderr:command terminated with exit code 22 2022-03-20T14:55:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:55:40.296 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:40.296764 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:55:40.296 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:40.296794 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:55:40.783 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:40.783339 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:55:40.783 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:40.783362 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:55:41.507 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:41.507175 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:55:41.507 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:41.507284 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:55:41.937 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:41.937636 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:55:41.937 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:41.937670 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-20T14:55:42.347 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:42.348072 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-20T14:55:42.347 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:42.348103 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T14:55:42.561 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:42.561739 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T14:55:42.561 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:42.561773 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T14:55:43.157 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:43.157540 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T14:55:43.581 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:43.582288 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T14:55:44.110 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:44.111034 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T14:55:44.940 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:44.940881 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T14:55:44.940 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:44.940905 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T14:55:45.481 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:45.481806 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T14:55:45.717 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:45.717420 I | cephclient: successfully enabled msgr2 protocol 2022-03-20T14:55:45.717 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:45.717457 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T14:55:45.717 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:45.717466 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T14:55:46.236 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:46.237001 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T14:55:46.287 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:46.287805 I | op-mgr: start running mgr 2022-03-20T14:55:46.287 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:46.287862 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T14:55:47.441 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:47.441108 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi134": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T14:55:50.206 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch status -f json 2022-03-20T14:55:50.780 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:55:50.781 INFO:teuthology.orchestra.run.smithi134.stderr:no valid command found; 10 closest matches: 2022-03-20T14:55:50.781 INFO:teuthology.orchestra.run.smithi134.stderr:pg stat 2022-03-20T14:55:50.781 INFO:teuthology.orchestra.run.smithi134.stderr:pg getmap 2022-03-20T14:55:50.781 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:55:50.782 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:55:50.782 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_pools_json 2022-03-20T14:55:50.782 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-pool [...] 2022-03-20T14:55:50.782 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-primary [] [...] 2022-03-20T14:55:50.783 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-osd [] [...] 2022-03-20T14:55:50.783 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls [] [...] 2022-03-20T14:55:50.783 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:55:50.783 INFO:teuthology.orchestra.run.smithi134.stderr:Error EINVAL: invalid command 2022-03-20T14:55:50.784 INFO:teuthology.orchestra.run.smithi134.stderr:command terminated with exit code 22 2022-03-20T14:55:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:55:56.115 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.116008 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-20T14:55:56.119 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.119798 I | op-mgr: setting services to point to mgr "a" 2022-03-20T14:55:56.187 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.188401 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T14:55:56.190 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.188751 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-20T14:55:56.190 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.188769 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T14:55:56.198 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.199031 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T14:55:56.202 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.203118 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-20T14:55:56.205 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.203473 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T14:55:56.205 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.203609 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T14:55:56.453 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.454171 W | cephclient: failed to enable mgr module "rook". trying again... 2022-03-20T14:55:56.484 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.485455 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T14:55:56.484 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.485475 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:55:56.493 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.494562 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-20T14:55:56.496 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.494597 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-20T14:55:56.497 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.494621 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-20T14:55:56.527 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.528967 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-20T14:55:56.528 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.528997 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-20T14:55:56.530 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.531155 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Normal:ReconcileSucceeded:successfully configured CephCluster "rook-ceph/rook-ceph" 2022-03-20T14:55:56.530 INFO:tasks.rook.operator.smithi134.stdout:I0320 14:55:56.531386 7 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-20T14:55:56.736 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:56.737256 I | op-mgr: successful modules: dashboard 2022-03-20T14:55:57.034 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:57.035662 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-20T14:55:57.764 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:55:57.765230 I | op-mgr: successful modules: prometheus 2022-03-20T14:56:00.788 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch status -f json 2022-03-20T14:56:01.200 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:56:01.200 INFO:teuthology.orchestra.run.smithi134.stderr:no valid command found; 10 closest matches: 2022-03-20T14:56:01.201 INFO:teuthology.orchestra.run.smithi134.stderr:pg stat 2022-03-20T14:56:01.201 INFO:teuthology.orchestra.run.smithi134.stderr:pg getmap 2022-03-20T14:56:01.201 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:56:01.201 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:56:01.202 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_pools_json 2022-03-20T14:56:01.202 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-pool [...] 2022-03-20T14:56:01.202 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-primary [] [...] 2022-03-20T14:56:01.202 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-osd [] [...] 2022-03-20T14:56:01.203 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls [] [...] 2022-03-20T14:56:01.203 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:56:01.203 INFO:teuthology.orchestra.run.smithi134.stderr:Error EINVAL: invalid command 2022-03-20T14:56:01.204 INFO:teuthology.orchestra.run.smithi134.stderr:command terminated with exit code 22 2022-03-20T14:56:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:56:01.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:01.865783 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T14:56:02.332 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:02.333044 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-20T14:56:03.258 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:03.259835 I | cephclient: command failed for set rook backend. trying again... 2022-03-20T14:56:07.622 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:07.623419 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-20T14:56:08.544 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:08.545090 I | cephclient: command failed for set rook backend. trying again... 2022-03-20T14:56:11.208 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch status -f json 2022-03-20T14:56:11.705 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:56:11.705 INFO:teuthology.orchestra.run.smithi134.stderr:no valid command found; 10 closest matches: 2022-03-20T14:56:11.706 INFO:teuthology.orchestra.run.smithi134.stderr:pg stat 2022-03-20T14:56:11.706 INFO:teuthology.orchestra.run.smithi134.stderr:pg getmap 2022-03-20T14:56:11.706 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:56:11.707 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:56:11.707 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_pools_json 2022-03-20T14:56:11.707 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-pool [...] 2022-03-20T14:56:11.707 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-primary [] [...] 2022-03-20T14:56:11.707 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls-by-osd [] [...] 2022-03-20T14:56:11.708 INFO:teuthology.orchestra.run.smithi134.stderr:pg ls [] [...] 2022-03-20T14:56:11.708 INFO:teuthology.orchestra.run.smithi134.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:56:11.708 INFO:teuthology.orchestra.run.smithi134.stderr:Error EINVAL: invalid command 2022-03-20T14:56:11.709 INFO:teuthology.orchestra.run.smithi134.stderr:command terminated with exit code 22 2022-03-20T14:56:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:56:12.921 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:12.923009 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-03-20T14:56:13.837 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:13.838495 I | cephclient: command failed for set rook backend. trying again... 2022-03-20T14:56:18.400 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:18.401696 I | op-mgr: successful modules: balancer 2022-03-20T14:56:19.292 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:19.293511 I | cephclient: action set rook backend succeeded on attempt 3 2022-03-20T14:56:19.292 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:19.293540 I | op-mgr: successful modules: orchestrator modules 2022-03-20T14:56:21.714 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch status -f json 2022-03-20T14:56:22.275 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:56:22.275 INFO:teuthology.orchestra.run.smithi134.stdout:{"available": true, "backend": "rook"} 2022-03-20T14:56:22.294 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-20T14:56:22.294 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-20T14:56:22.918 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch apply osd --all-available-devices 2022-03-20T14:56:23.554 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:23.555280 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-20T14:56:23.555 INFO:tasks.rook.operator.smithi134.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-20T14:56:23.555 INFO:tasks.rook.operator.smithi134.stdout: Storage: v1.StorageScopeSpec{ 2022-03-20T14:56:23.555 INFO:tasks.rook.operator.smithi134.stdout: ... // 3 identical fields 2022-03-20T14:56:23.555 INFO:tasks.rook.operator.smithi134.stdout: Config: nil, 2022-03-20T14:56:23.556 INFO:tasks.rook.operator.smithi134.stdout: Selection: {}, 2022-03-20T14:56:23.556 INFO:tasks.rook.operator.smithi134.stdout:- StorageClassDeviceSets: nil, 2022-03-20T14:56:23.556 INFO:tasks.rook.operator.smithi134.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-20T14:56:23.556 INFO:tasks.rook.operator.smithi134.stdout:+ { 2022-03-20T14:56:23.557 INFO:tasks.rook.operator.smithi134.stdout:+ Name: "smithi134-lv-1", 2022-03-20T14:56:23.557 INFO:tasks.rook.operator.smithi134.stdout:+ Count: 1, 2022-03-20T14:56:23.557 INFO:tasks.rook.operator.smithi134.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:56:23.557 INFO:tasks.rook.operator.smithi134.stdout:+ }, 2022-03-20T14:56:23.559 INFO:tasks.rook.operator.smithi134.stdout:+ { 2022-03-20T14:56:23.559 INFO:tasks.rook.operator.smithi134.stdout:+ Name: "smithi134-lv-2", 2022-03-20T14:56:23.560 INFO:tasks.rook.operator.smithi134.stdout:+ Count: 1, 2022-03-20T14:56:23.560 INFO:tasks.rook.operator.smithi134.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:56:23.560 INFO:tasks.rook.operator.smithi134.stdout:+ }, 2022-03-20T14:56:23.560 INFO:tasks.rook.operator.smithi134.stdout:+ { 2022-03-20T14:56:23.561 INFO:tasks.rook.operator.smithi134.stdout:+ Name: "smithi134-lv-3", 2022-03-20T14:56:23.561 INFO:tasks.rook.operator.smithi134.stdout:+ Count: 1, 2022-03-20T14:56:23.561 INFO:tasks.rook.operator.smithi134.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:56:23.561 INFO:tasks.rook.operator.smithi134.stdout:+ }, 2022-03-20T14:56:23.561 INFO:tasks.rook.operator.smithi134.stdout:+ { 2022-03-20T14:56:23.562 INFO:tasks.rook.operator.smithi134.stdout:+ Name: "smithi134-lv-4", 2022-03-20T14:56:23.562 INFO:tasks.rook.operator.smithi134.stdout:+ Count: 1, 2022-03-20T14:56:23.562 INFO:tasks.rook.operator.smithi134.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:56:23.562 INFO:tasks.rook.operator.smithi134.stdout:+ }, 2022-03-20T14:56:23.563 INFO:tasks.rook.operator.smithi134.stdout:+ }, 2022-03-20T14:56:23.563 INFO:tasks.rook.operator.smithi134.stdout: }, 2022-03-20T14:56:23.563 INFO:tasks.rook.operator.smithi134.stdout: Annotations: nil, 2022-03-20T14:56:23.563 INFO:tasks.rook.operator.smithi134.stdout: Labels: nil, 2022-03-20T14:56:23.563 INFO:tasks.rook.operator.smithi134.stdout: ... // 20 identical fields 2022-03-20T14:56:23.564 INFO:tasks.rook.operator.smithi134.stdout: } 2022-03-20T14:56:23.564 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:23.555382 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:56:23.564 INFO:teuthology.orchestra.run.smithi134.stdout:['Success'] 2022-03-20T14:56:23.565 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:23.562786 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T14:56:23.578 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:23.579739 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T14:56:23.585 INFO:tasks.rook:Waiting for 4 OSDs 2022-03-20T14:56:23.586 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:56:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:56:24.154 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":4,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:56:24.154 INFO:tasks.rook: have 0/4 OSDs 2022-03-20T14:56:28.860 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:28.861397 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T14:56:28.860 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:28.861417 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T14:56:28.860 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:28.861426 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11187 quincy", pursuing 2022-03-20T14:56:28.870 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:28.870976 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T14:56:28.881 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:28.874000 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:56:28.881 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:28.874523 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:56:29.286 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:29.287965 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T14:56:29.549 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:29.550624 I | op-mon: start running mons 2022-03-20T14:56:29.664 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:29.664513 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T14:56:29.714 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:29.716271 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T14:56:29.722 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:29.724770 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:56:29.725 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:29.724980 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:56:30.833 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:30.834982 I | op-mon: targeting the mon count 1 2022-03-20T14:56:30.847 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:30.849033 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-20T14:56:31.848 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:31.850224 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:56:31.855 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:31.857532 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T14:56:31.871 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:31.872792 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T14:56:31.897 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:31.898674 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-20T14:56:31.902 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:31.904510 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-20T14:56:33.909 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:33.910645 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-20T14:56:34.156 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:56:34.655 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:56:34.671 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":4,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:56:34.672 INFO:tasks.rook: have 0/4 OSDs 2022-03-20T14:56:39.717 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:39.719100 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T14:56:39.718 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:39.719126 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T14:56:39.719 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:39.719137 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11187 quincy", pursuing 2022-03-20T14:56:39.785 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:39.786810 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T14:56:40.173 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:40.172190 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:56:40.176 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:40.175164 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:56:40.570 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:40.572811 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T14:56:40.668 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:40.667582 I | op-mon: start running mons 2022-03-20T14:56:40.672 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:40.674710 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T14:56:40.741 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:40.742737 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T14:56:40.748 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:40.750403 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:56:40.748 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:40.750566 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:56:41.742 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:41.744368 I | op-mon: targeting the mon count 1 2022-03-20T14:56:41.747 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:41.749043 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:56:42.118 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:42.120661 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:56:42.119 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:42.120690 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:56:42.445 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:42.446923 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:56:42.445 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:42.446947 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:56:42.776 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:42.778449 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:56:42.776 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:42.778477 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:56:43.142 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.144007 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:56:43.142 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.144035 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-20T14:56:43.472 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.473819 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-20T14:56:43.472 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.473851 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T14:56:43.822 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.823927 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T14:56:43.822 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.823958 I | op-mon: checking for basic quorum with existing mons 2022-03-20T14:56:43.822 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.823966 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-20T14:56:43.834 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.836869 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T14:56:43.841 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.843502 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:56:43.841 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.843698 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:56:43.853 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.855547 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T14:56:43.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.866257 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T14:56:43.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.866287 I | op-mon: waiting for mon quorum with [a] 2022-03-20T14:56:43.868 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:43.870570 I | op-mon: mons running: [a] 2022-03-20T14:56:44.309 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:44.311396 I | op-mon: Monitors in quorum: [a] 2022-03-20T14:56:44.310 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:44.311430 I | op-mon: mons created: 1 2022-03-20T14:56:44.673 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:56:44.709 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:44.712083 I | op-mon: waiting for mon quorum with [a] 2022-03-20T14:56:44.716 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:44.718318 I | op-mon: mons running: [a] 2022-03-20T14:56:45.177 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:45.178992 I | op-mon: Monitors in quorum: [a] 2022-03-20T14:56:45.177 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:45.179028 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T14:56:45.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:56:45.287 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":4,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:56:45.288 INFO:tasks.rook: have 0/4 OSDs 2022-03-20T14:56:45.601 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:45.603878 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T14:56:45.619 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:45.621852 I | op-mon: checking if multiple mons are on the same node 2022-03-20T14:56:46.011 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:46.013344 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T14:56:46.473 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:46.475423 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T14:56:46.916 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:46.918283 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T14:56:46.916 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:46.918305 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T14:56:47.377 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:47.378905 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T14:56:47.786 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:47.788000 I | cephclient: successfully enabled msgr2 protocol 2022-03-20T14:56:47.786 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:47.788024 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T14:56:47.786 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:47.788030 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T14:56:48.192 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.193987 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T14:56:48.245 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.246967 I | op-mgr: start running mgr 2022-03-20T14:56:48.245 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.246997 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T14:56:48.711 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.712810 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T14:56:48.716 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.718564 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T14:56:48.719 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.720689 I | op-mgr: setting services to point to mgr "a" 2022-03-20T14:56:48.850 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.851979 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T14:56:48.850 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.852058 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T14:56:48.870 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.873257 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T14:56:48.901 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:48.903697 I | op-osd: creating 1 new PVCs for device set "smithi134-lv-1" 2022-03-20T14:56:49.107 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.110050 I | op-osd: successfully provisioned PVC "smithi134-lv-1-data-02g9hz" 2022-03-20T14:56:49.108 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.110081 I | op-osd: creating 1 new PVCs for device set "smithi134-lv-2" 2022-03-20T14:56:49.342 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.344843 I | op-osd: successfully provisioned PVC "smithi134-lv-2-data-0vt7mf" 2022-03-20T14:56:49.347 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.345181 I | op-osd: creating 1 new PVCs for device set "smithi134-lv-3" 2022-03-20T14:56:49.504 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.506120 I | op-osd: successfully provisioned PVC "smithi134-lv-3-data-0nd6cl" 2022-03-20T14:56:49.542 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.506280 I | op-osd: creating 1 new PVCs for device set "smithi134-lv-4" 2022-03-20T14:56:49.734 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.736381 I | op-osd: successfully provisioned PVC "smithi134-lv-4-data-0dmsjd" 2022-03-20T14:56:49.737 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.739451 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T14:56:49.737 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.739499 I | op-mgr: successful modules: dashboard 2022-03-20T14:56:49.739 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.741207 I | op-mgr: successful modules: prometheus 2022-03-20T14:56:49.753 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:49.756235 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-1-data-02g9hz" 2022-03-20T14:56:50.114 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:50.116369 I | op-osd: started OSD provisioning job for PVC "smithi134-lv-1-data-02g9hz" 2022-03-20T14:56:50.115 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:50.116397 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-2-data-0vt7mf" 2022-03-20T14:56:50.125 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:50.128253 I | op-mgr: successful modules: balancer 2022-03-20T14:56:50.515 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:50.516730 I | op-osd: started OSD provisioning job for PVC "smithi134-lv-2-data-0vt7mf" 2022-03-20T14:56:50.515 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:50.516760 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-3-data-0nd6cl" 2022-03-20T14:56:50.981 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:50.983271 I | op-osd: started OSD provisioning job for PVC "smithi134-lv-3-data-0nd6cl" 2022-03-20T14:56:50.981 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:50.983303 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-4-data-0dmsjd" 2022-03-20T14:56:51.095 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:51.097465 I | op-mgr: successful modules: orchestrator modules 2022-03-20T14:56:51.522 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:51.524702 I | op-osd: started OSD provisioning job for PVC "smithi134-lv-4-data-0dmsjd" 2022-03-20T14:56:51.523 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:51.524726 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T14:56:51.523 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:51.524732 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T14:56:51.702 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:51.704130 I | op-osd: OSD orchestration status for PVC smithi134-lv-1-data-02g9hz is "starting" 2022-03-20T14:56:51.703 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:51.704308 I | op-osd: OSD orchestration status for PVC smithi134-lv-2-data-0vt7mf is "starting" 2022-03-20T14:56:51.703 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:51.704369 I | op-osd: OSD orchestration status for PVC smithi134-lv-3-data-0nd6cl is "starting" 2022-03-20T14:56:51.703 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:56:51.704450 I | op-osd: OSD orchestration status for PVC smithi134-lv-4-data-0dmsjd is "starting" 2022-03-20T14:56:55.289 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:56:56.336 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:56:56.353 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:56:56.353 INFO:tasks.rook: have 0/4 OSDs 2022-03-20T14:57:03.904 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:03.906571 I | op-osd: OSD orchestration status for node smithi134-lv-4-data-0dmsjd is "orchestrating" 2022-03-20T14:57:04.180 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:04.182567 I | op-osd: OSD orchestration status for node smithi134-lv-2-data-0vt7mf is "orchestrating" 2022-03-20T14:57:04.896 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:04.892421 I | op-osd: OSD orchestration status for PVC smithi134-lv-4-data-0dmsjd is "orchestrating" 2022-03-20T14:57:04.926 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:04.928669 I | op-osd: OSD orchestration status for PVC smithi134-lv-2-data-0vt7mf is "orchestrating" 2022-03-20T14:57:06.354 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:57:06.803 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:57:06.820 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:57:06.820 INFO:tasks.rook: have 0/4 OSDs 2022-03-20T14:57:11.823 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:11.825729 I | op-osd: OSD orchestration status for node smithi134-lv-3-data-0nd6cl is "orchestrating" 2022-03-20T14:57:11.885 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:11.888428 I | op-osd: OSD orchestration status for node smithi134-lv-1-data-02g9hz is "orchestrating" 2022-03-20T14:57:11.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:11.975494 I | op-osd: OSD orchestration status for PVC smithi134-lv-4-data-0dmsjd is "completed" 2022-03-20T14:57:11.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:11.975524 I | op-osd: creating OSD 0 on PVC "smithi134-lv-4-data-0dmsjd" 2022-03-20T14:57:11.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:11.975535 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-4-data-0dmsjd" 2022-03-20T14:57:12.686 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:12.687729 I | op-osd: OSD orchestration status for PVC smithi134-lv-2-data-0vt7mf is "completed" 2022-03-20T14:57:12.690 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:12.692478 I | op-osd: creating OSD 1 on PVC "smithi134-lv-2-data-0vt7mf" 2022-03-20T14:57:12.690 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:12.692695 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-2-data-0vt7mf" 2022-03-20T14:57:13.484 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:13.477771 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi134": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T14:57:14.197 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:14.057968 I | op-osd: OSD orchestration status for PVC smithi134-lv-3-data-0nd6cl is "orchestrating" 2022-03-20T14:57:14.197 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:14.058796 I | op-osd: OSD orchestration status for PVC smithi134-lv-1-data-02g9hz is "orchestrating" 2022-03-20T14:57:16.822 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:57:17.425 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:57:17.446 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":9,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1647788235,"num_remapped_pgs":0} 2022-03-20T14:57:17.446 INFO:tasks.rook: have 0/4 OSDs 2022-03-20T14:57:20.342 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:20.344811 I | op-osd: OSD orchestration status for PVC smithi134-lv-3-data-0nd6cl is "completed" 2022-03-20T14:57:20.378 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:20.344842 I | op-osd: creating OSD 2 on PVC "smithi134-lv-3-data-0nd6cl" 2022-03-20T14:57:20.379 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:20.344855 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-3-data-0nd6cl" 2022-03-20T14:57:21.931 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:21.934176 I | op-osd: OSD orchestration status for PVC smithi134-lv-1-data-02g9hz is "completed" 2022-03-20T14:57:21.934 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:21.934610 I | op-osd: creating OSD 3 on PVC "smithi134-lv-1-data-02g9hz" 2022-03-20T14:57:21.935 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:21.934632 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-1-data-02g9hz" 2022-03-20T14:57:25.382 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:25.385544 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T14:57:25.383 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:25.385567 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:57:27.448 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:57:28.417 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:57:28.434 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":9,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1647788235,"num_remapped_pgs":0} 2022-03-20T14:57:28.434 INFO:tasks.rook: have 0/4 OSDs 2022-03-20T14:57:33.999 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 14:57:34.001135 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi134": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T14:57:38.436 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:57:39.229 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:57:39.247 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":9,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1647788235,"num_remapped_pgs":0} 2022-03-20T14:57:39.248 INFO:tasks.rook: have 0/4 OSDs 2022-03-20T14:57:49.249 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:57:50.619 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:57:50.643 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":12,"num_osds":4,"num_up_osds":1,"osd_up_since":1647788269,"num_in_osds":4,"osd_in_since":1647788235,"num_remapped_pgs":0} 2022-03-20T14:57:50.643 INFO:tasks.rook: have 1/4 OSDs 2022-03-20T14:58:00.645 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd stat -f json 2022-03-20T14:58:01.135 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:01.153 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":21,"num_osds":4,"num_up_osds":4,"osd_up_since":1647788277,"num_in_osds":4,"osd_in_since":1647788235,"num_remapped_pgs":0} 2022-03-20T14:58:01.153 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-20T14:58:01.154 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- cat /etc/ceph/ceph.conf 2022-03-20T14:58:01.306 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2022-03-20T14:58:01.306 INFO:teuthology.orchestra.run.smithi134.stdout:mon_host = 172.21.15.134:6789 2022-03-20T14:58:01.307 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:01.307 INFO:teuthology.orchestra.run.smithi134.stdout:[client.admin] 2022-03-20T14:58:01.307 INFO:teuthology.orchestra.run.smithi134.stdout:keyring = /etc/ceph/keyring 2022-03-20T14:58:01.317 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- cat /etc/ceph/keyring 2022-03-20T14:58:01.478 INFO:teuthology.orchestra.run.smithi134.stdout:[client.admin] 2022-03-20T14:58:01.479 INFO:teuthology.orchestra.run.smithi134.stdout:key = AQBHQDdiMKK4FBAAqXlfc9NRiLdmJQpJFQS2PQ== 2022-03-20T14:58:01.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2022-03-20T14:58:01.492 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:58:01.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.conf 2022-03-20T14:58:01.502 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:58:01.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/keyring 2022-03-20T14:58:01.512 INFO:tasks.rook:Setting up client nodes... 2022-03-20T14:58:01.513 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-20T14:58:02.185 INFO:teuthology.orchestra.run.smithi134.stdout:[client.a] 2022-03-20T14:58:02.185 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD6QDdiAyfGBxAA+0kke/szRX1Mb8OBh0F6oA== 2022-03-20T14:58:02.205 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:58:02.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-20T14:58:02.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-20T14:58:02.224 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd dump --format=json 2022-03-20T14:58:02.733 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:02.734 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":22,"fsid":"28430573-5abd-4162-9275-e2a8283a7c3a","created":"2022-03-20T14:55:38.009118+0000","modified":"2022-03-20T14:58:01.886604+0000","last_up_change":"2022-03-20T14:57:57.134635+0000","last_in_change":"2022-03-20T14:57:15.664298+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-20T14:57:59.236224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"89dfaf99-7a2c-43e2-a400-bb0d41a5dce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6803","nonce":2356650006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6805","nonce":2356650006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6809","nonce":2356650006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6807","nonce":2356650006}]},"public_addr":"172.21.15.134:6803/2356650006","cluster_addr":"172.21.15.134:6805/2356650006","heartbeat_back_addr":"172.21.15.134:6809/2356650006","heartbeat_front_addr":"172.21.15.134:6807/2356650006","state":["exists","up"]},{"osd":1,"uuid":"f3a0f663-c591-4085-950f-cc846ecc1adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6811","nonce":2291082527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6813","nonce":2291082527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6817","nonce":2291082527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6815","nonce":2291082527}]},"public_addr":"172.21.15.134:6811/2291082527","cluster_addr":"172.21.15.134:6813/2291082527","heartbeat_back_addr":"172.21.15.134:6817/2291082527","heartbeat_front_addr":"172.21.15.134:6815/2291082527","state":["exists","up"]},{"osd":2,"uuid":"f3a524f6-e64b-43e4-8eb7-1139b143384c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6819","nonce":1817821717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6821","nonce":1817821717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6825","nonce":1817821717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6823","nonce":1817821717}]},"public_addr":"172.21.15.134:6819/1817821717","cluster_addr":"172.21.15.134:6821/1817821717","heartbeat_back_addr":"172.21.15.134:6825/1817821717","heartbeat_front_addr":"172.21.15.134:6823/1817821717","state":["exists","up"]},{"osd":3,"uuid":"80c97fd4-e7cd-4f71-978f-d6dbfaf25500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6827","nonce":4081235876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6829","nonce":4081235876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6833","nonce":4081235876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6831","nonce":4081235876}]},"public_addr":"172.21.15.134:6827/4081235876","cluster_addr":"172.21.15.134:6829/4081235876","heartbeat_back_addr":"172.21.15.134:6833/4081235876","heartbeat_front_addr":"172.21.15.134:6831/4081235876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:45.946178+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:52.912611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:55.389423+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:55.520670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-20T14:58:02.750 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-20T14:57:59.236224+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-20T14:58:02.751 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd pool get .mgr pg_num 2022-03-20T14:58:03.246 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 1 2022-03-20T14:58:03.265 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-20T14:58:03.266 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-20T14:58:03.267 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph mgr dump --format=json 2022-03-20T14:58:03.885 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:03.901 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":12,"active_gid":4168,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":1503165754},{"type":"v1","addr":"172.21.15.134:6801","nonce":1503165754}]},"active_addr":"172.21.15.134:6801/1503165754","active_change":"2022-03-20T14:56:14.582287+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://172.21.15.134:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":2625295663}]},{"addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":2931976054}]},{"addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1725567480}]}]}} 2022-03-20T14:58:03.904 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-20T14:58:03.905 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-20T14:58:03.905 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd dump --format=json 2022-03-20T14:58:04.405 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:04.405 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":22,"fsid":"28430573-5abd-4162-9275-e2a8283a7c3a","created":"2022-03-20T14:55:38.009118+0000","modified":"2022-03-20T14:58:01.886604+0000","last_up_change":"2022-03-20T14:57:57.134635+0000","last_in_change":"2022-03-20T14:57:15.664298+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-20T14:57:59.236224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"89dfaf99-7a2c-43e2-a400-bb0d41a5dce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6803","nonce":2356650006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6805","nonce":2356650006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6809","nonce":2356650006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6807","nonce":2356650006}]},"public_addr":"172.21.15.134:6803/2356650006","cluster_addr":"172.21.15.134:6805/2356650006","heartbeat_back_addr":"172.21.15.134:6809/2356650006","heartbeat_front_addr":"172.21.15.134:6807/2356650006","state":["exists","up"]},{"osd":1,"uuid":"f3a0f663-c591-4085-950f-cc846ecc1adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6811","nonce":2291082527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6813","nonce":2291082527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6817","nonce":2291082527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6815","nonce":2291082527}]},"public_addr":"172.21.15.134:6811/2291082527","cluster_addr":"172.21.15.134:6813/2291082527","heartbeat_back_addr":"172.21.15.134:6817/2291082527","heartbeat_front_addr":"172.21.15.134:6815/2291082527","state":["exists","up"]},{"osd":2,"uuid":"f3a524f6-e64b-43e4-8eb7-1139b143384c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6819","nonce":1817821717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6821","nonce":1817821717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6825","nonce":1817821717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6823","nonce":1817821717}]},"public_addr":"172.21.15.134:6819/1817821717","cluster_addr":"172.21.15.134:6821/1817821717","heartbeat_back_addr":"172.21.15.134:6825/1817821717","heartbeat_front_addr":"172.21.15.134:6823/1817821717","state":["exists","up"]},{"osd":3,"uuid":"80c97fd4-e7cd-4f71-978f-d6dbfaf25500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6827","nonce":4081235876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6829","nonce":4081235876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6833","nonce":4081235876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6831","nonce":4081235876}]},"public_addr":"172.21.15.134:6827/4081235876","cluster_addr":"172.21.15.134:6829/4081235876","heartbeat_back_addr":"172.21.15.134:6833/4081235876","heartbeat_front_addr":"172.21.15.134:6831/4081235876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:45.946178+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:52.912611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:55.389423+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:55.520670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-20T14:58:04.425 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-20T14:58:04.425 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd dump --format=json 2022-03-20T14:58:04.926 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:04.926 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":22,"fsid":"28430573-5abd-4162-9275-e2a8283a7c3a","created":"2022-03-20T14:55:38.009118+0000","modified":"2022-03-20T14:58:01.886604+0000","last_up_change":"2022-03-20T14:57:57.134635+0000","last_in_change":"2022-03-20T14:57:15.664298+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-20T14:57:59.236224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"89dfaf99-7a2c-43e2-a400-bb0d41a5dce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6803","nonce":2356650006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6805","nonce":2356650006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6809","nonce":2356650006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2356650006},{"type":"v1","addr":"172.21.15.134:6807","nonce":2356650006}]},"public_addr":"172.21.15.134:6803/2356650006","cluster_addr":"172.21.15.134:6805/2356650006","heartbeat_back_addr":"172.21.15.134:6809/2356650006","heartbeat_front_addr":"172.21.15.134:6807/2356650006","state":["exists","up"]},{"osd":1,"uuid":"f3a0f663-c591-4085-950f-cc846ecc1adb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6811","nonce":2291082527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6813","nonce":2291082527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6817","nonce":2291082527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":2291082527},{"type":"v1","addr":"172.21.15.134:6815","nonce":2291082527}]},"public_addr":"172.21.15.134:6811/2291082527","cluster_addr":"172.21.15.134:6813/2291082527","heartbeat_back_addr":"172.21.15.134:6817/2291082527","heartbeat_front_addr":"172.21.15.134:6815/2291082527","state":["exists","up"]},{"osd":2,"uuid":"f3a524f6-e64b-43e4-8eb7-1139b143384c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6819","nonce":1817821717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6821","nonce":1817821717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6825","nonce":1817821717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":1817821717},{"type":"v1","addr":"172.21.15.134:6823","nonce":1817821717}]},"public_addr":"172.21.15.134:6819/1817821717","cluster_addr":"172.21.15.134:6821/1817821717","heartbeat_back_addr":"172.21.15.134:6825/1817821717","heartbeat_front_addr":"172.21.15.134:6823/1817821717","state":["exists","up"]},{"osd":3,"uuid":"80c97fd4-e7cd-4f71-978f-d6dbfaf25500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6827","nonce":4081235876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6829","nonce":4081235876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6833","nonce":4081235876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":4081235876},{"type":"v1","addr":"172.21.15.134:6831","nonce":4081235876}]},"public_addr":"172.21.15.134:6827/4081235876","cluster_addr":"172.21.15.134:6829/4081235876","heartbeat_back_addr":"172.21.15.134:6833/4081235876","heartbeat_front_addr":"172.21.15.134:6831/4081235876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:45.946178+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:52.912611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:55.389423+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:57:55.520670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-20T14:58:04.946 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph tell osd.0 flush_pg_stats 2022-03-20T14:58:04.947 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph tell osd.1 flush_pg_stats 2022-03-20T14:58:04.948 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph tell osd.2 flush_pg_stats 2022-03-20T14:58:04.948 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph tell osd.3 flush_pg_stats 2022-03-20T14:58:05.366 INFO:teuthology.orchestra.run.smithi134.stdout:77309411333 2022-03-20T14:58:05.367 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd last-stat-seq osd.2 2022-03-20T14:58:05.456 INFO:teuthology.orchestra.run.smithi134.stdout:77309411333 2022-03-20T14:58:05.457 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd last-stat-seq osd.3 2022-03-20T14:58:05.475 INFO:teuthology.orchestra.run.smithi134.stdout:68719476740 2022-03-20T14:58:05.476 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd last-stat-seq osd.1 2022-03-20T14:58:05.487 INFO:teuthology.orchestra.run.smithi134.stdout:51539607558 2022-03-20T14:58:05.488 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd last-stat-seq osd.0 2022-03-20T14:58:05.961 INFO:teuthology.orchestra.run.smithi134.stdout:77309411332 2022-03-20T14:58:05.978 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411333 got 77309411332 for osd.2 2022-03-20T14:58:06.019 INFO:teuthology.orchestra.run.smithi134.stdout:68719476739 2022-03-20T14:58:06.032 INFO:teuthology.orchestra.run.smithi134.stdout:77309411332 2022-03-20T14:58:06.038 INFO:tasks.rook.ceph_manager.ceph:need seq 68719476740 got 68719476739 for osd.1 2022-03-20T14:58:06.040 INFO:teuthology.orchestra.run.smithi134.stdout:51539607556 2022-03-20T14:58:06.063 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411333 got 77309411332 for osd.3 2022-03-20T14:58:06.065 INFO:tasks.rook.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.0 2022-03-20T14:58:06.980 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd last-stat-seq osd.2 2022-03-20T14:58:07.038 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd last-stat-seq osd.1 2022-03-20T14:58:07.064 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd last-stat-seq osd.3 2022-03-20T14:58:07.066 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd last-stat-seq osd.0 2022-03-20T14:58:07.572 INFO:teuthology.orchestra.run.smithi134.stdout:68719476741 2022-03-20T14:58:07.591 INFO:tasks.rook.ceph_manager.ceph:need seq 68719476740 got 68719476741 for osd.1 2022-03-20T14:58:07.591 DEBUG:teuthology.parallel:result is None 2022-03-20T14:58:07.667 INFO:teuthology.orchestra.run.smithi134.stdout:77309411333 2022-03-20T14:58:07.686 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411333 got 77309411333 for osd.2 2022-03-20T14:58:07.686 DEBUG:teuthology.parallel:result is None 2022-03-20T14:58:07.689 INFO:teuthology.orchestra.run.smithi134.stdout:51539607558 2022-03-20T14:58:07.711 INFO:tasks.rook.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.0 2022-03-20T14:58:07.711 DEBUG:teuthology.parallel:result is None 2022-03-20T14:58:07.763 INFO:teuthology.orchestra.run.smithi134.stdout:77309411333 2022-03-20T14:58:07.781 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411333 got 77309411333 for osd.3 2022-03-20T14:58:07.782 DEBUG:teuthology.parallel:result is None 2022-03-20T14:58:07.782 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-20T14:58:07.782 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph pg dump --format=json 2022-03-20T14:58:08.297 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:08.300 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2022-03-20T14:58:08.317 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2022-03-20T14:58:06.758571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23580,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":20544,"kb_avail":374957028,"statfs":{"total":383980142592,"available":383955996672,"internally_reserved":0,"allocated":2035712,"data_stored":1461610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21037056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"22'97","reported_seq":137,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-03-20T14:58:05.441061+0000","last_change":"2022-03-20T14:58:01.296610+0000","last_active":"2022-03-20T14:58:05.441061+0000","last_peered":"2022-03-20T14:58:05.441061+0000","last_clean":"2022-03-20T14:58:05.441061+0000","last_became_active":"2022-03-20T14:58:01.294433+0000","last_became_peered":"2022-03-20T14:58:01.294433+0000","last_unstale":"2022-03-20T14:58:05.441061+0000","last_undegraded":"2022-03-20T14:58:05.441061+0000","last_fullsized":"2022-03-20T14:58:05.441061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-20T14:57:59.464972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-20T14:57:59.464972+0000","last_clean_scrub_stamp":"2022-03-20T14:57:59.464972+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-21T23:16:10.121827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1351680,"data_stored":1339428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":18,"seq":77309411333,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5048,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740104,"statfs":{"total":95995035648,"available":95989866496,"internally_reserved":0,"allocated":180224,"data_stored":43246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5329999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411333,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6092,"kb_used_data":584,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739060,"statfs":{"total":95995035648,"available":95988797440,"internally_reserved":0,"allocated":598016,"data_stored":452308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]}]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":584,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":598016,"data_stored":452308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6284,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738868,"statfs":{"total":95995035648,"available":95988600832,"internally_reserved":0,"allocated":659456,"data_stored":513748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-20T14:58:08.318 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph pg dump --format=json 2022-03-20T14:58:08.828 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:08.830 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2022-03-20T14:58:08.851 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2022-03-20T14:58:08.759144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23524,"kb_used_data":2056,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":374957084,"statfs":{"total":383980142592,"available":383956054016,"internally_reserved":0,"allocated":2105344,"data_stored":1526596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000573"},"pg_stats":[{"pgid":"1.0","version":"22'97","reported_seq":137,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-03-20T14:58:05.441061+0000","last_change":"2022-03-20T14:58:01.296610+0000","last_active":"2022-03-20T14:58:05.441061+0000","last_peered":"2022-03-20T14:58:05.441061+0000","last_clean":"2022-03-20T14:58:05.441061+0000","last_became_active":"2022-03-20T14:58:01.294433+0000","last_became_peered":"2022-03-20T14:58:01.294433+0000","last_unstale":"2022-03-20T14:58:05.441061+0000","last_undegraded":"2022-03-20T14:58:05.441061+0000","last_fullsized":"2022-03-20T14:58:05.441061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-20T14:57:59.464972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-20T14:57:59.464972+0000","last_clean_scrub_stamp":"2022-03-20T14:57:59.464972+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-21T23:16:10.121827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":18,"seq":77309411334,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5056,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740096,"statfs":{"total":95995035648,"available":95989858304,"internally_reserved":0,"allocated":188416,"data_stored":46792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6028,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739124,"statfs":{"total":95995035648,"available":95988862976,"internally_reserved":0,"allocated":659456,"data_stored":513748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]}]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":584,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":598016,"data_stored":452308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6284,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738868,"statfs":{"total":95995035648,"available":95988600832,"internally_reserved":0,"allocated":659456,"data_stored":513748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-20T14:58:08.852 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-20T14:58:08.852 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-20T14:58:08.852 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-20T14:58:08.853 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph health --format=json 2022-03-20T14:58:09.396 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:09.396 INFO:teuthology.orchestra.run.smithi134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-20T14:58:09.415 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-20T14:58:09.416 INFO:tasks.rook:Rook complete, yielding 2022-03-20T14:58:09.416 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-20T14:58:09.429 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph -s 2022-03-20T14:58:10.007 INFO:teuthology.orchestra.run.smithi134.stdout: cluster: 2022-03-20T14:58:10.007 INFO:teuthology.orchestra.run.smithi134.stdout: id: 28430573-5abd-4162-9275-e2a8283a7c3a 2022-03-20T14:58:10.007 INFO:teuthology.orchestra.run.smithi134.stdout: health: HEALTH_OK 2022-03-20T14:58:10.008 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:10.008 INFO:teuthology.orchestra.run.smithi134.stdout: services: 2022-03-20T14:58:10.008 INFO:teuthology.orchestra.run.smithi134.stdout: mon: 1 daemons, quorum a (age 2m) 2022-03-20T14:58:10.008 INFO:teuthology.orchestra.run.smithi134.stdout: mgr: a(active, since 115s) 2022-03-20T14:58:10.009 INFO:teuthology.orchestra.run.smithi134.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 54s) 2022-03-20T14:58:10.009 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:10.009 INFO:teuthology.orchestra.run.smithi134.stdout: data: 2022-03-20T14:58:10.009 INFO:teuthology.orchestra.run.smithi134.stdout: pools: 1 pools, 1 pgs 2022-03-20T14:58:10.009 INFO:teuthology.orchestra.run.smithi134.stdout: objects: 2 objects, 577 KiB 2022-03-20T14:58:10.010 INFO:teuthology.orchestra.run.smithi134.stdout: usage: 23 MiB used, 358 GiB / 358 GiB avail 2022-03-20T14:58:10.010 INFO:teuthology.orchestra.run.smithi134.stdout: pgs: 1 active+clean 2022-03-20T14:58:10.010 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T14:58:10.030 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch status 2022-03-20T14:58:10.577 INFO:teuthology.orchestra.run.smithi134.stdout:Backend: rook 2022-03-20T14:58:10.577 INFO:teuthology.orchestra.run.smithi134.stdout:Available: Yes 2022-03-20T14:58:10.601 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ps 2022-03-20T14:58:11.109 INFO:teuthology.orchestra.run.smithi134.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-20T14:58:11.109 INFO:teuthology.orchestra.run.smithi134.stdout:crashcollector.crash smithi134 running (58s) 0s ago 59s - - 7c0c85ada2e4 2022-03-20T14:58:11.109 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.a smithi134 running (2m) 0s ago 2m - - 7c0c85ada2e4 2022-03-20T14:58:11.110 INFO:teuthology.orchestra.run.smithi134.stdout:mon.a smithi134 running (2m) 0s ago 2m - - 7c0c85ada2e4 2022-03-20T14:58:11.110 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0 smithi134 running (59s) 0s ago 59s - - 7c0c85ada2e4 2022-03-20T14:58:11.110 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1 smithi134 running (57s) 0s ago 57s - - 7c0c85ada2e4 2022-03-20T14:58:11.110 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2 smithi134 running (49s) 0s ago 50s - - 7c0c85ada2e4 2022-03-20T14:58:11.111 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3 smithi134 running (48s) 0s ago 48s - - 7c0c85ada2e4 2022-03-20T14:58:11.133 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls 2022-03-20T14:58:11.762 INFO:teuthology.orchestra.run.smithi134.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-20T14:58:11.762 INFO:teuthology.orchestra.run.smithi134.stdout:crash 1/1 0s ago 59s * 2022-03-20T14:58:11.762 INFO:teuthology.orchestra.run.smithi134.stdout:mgr 1/1 0s ago 2m count:1 2022-03-20T14:58:11.763 INFO:teuthology.orchestra.run.smithi134.stdout:mon 1/1 0s ago 2m count:1 2022-03-20T14:58:11.763 INFO:teuthology.orchestra.run.smithi134.stdout:osd 8 0s ago 59s 2022-03-20T14:58:11.763 INFO:teuthology.orchestra.run.smithi134.stdout:osd.all-available-devices 0 0s ago - * 2022-03-20T14:58:11.784 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch device ls 2022-03-20T14:58:12.297 INFO:teuthology.orchestra.run.smithi134.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-20T14:58:12.298 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T14:58:12.299 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T14:58:12.299 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T14:58:12.299 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T14:58:12.319 INFO:teuthology.run_tasks:Running task install... 2022-03-20T14:58:12.341 DEBUG:teuthology.task.install:project ceph 2022-03-20T14:58:12.342 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e'}} 2022-03-20T14:58:12.343 DEBUG:teuthology.task.install:config {'host.a': None, 'flavor': 'default', 'sha1': 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e'} 2022-03-20T14:58:12.344 INFO:teuthology.task.install:Using flavor: default 2022-03-20T14:58:12.354 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-03-20T14:58:12.355 INFO:teuthology.task.install:extra packages: [] 2022-03-20T14:58:12.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo lsb_release -is 2022-03-20T14:58:12.404 INFO:teuthology.orchestra.run.smithi134.stdout:Ubuntu 2022-03-20T14:58:12.409 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-20T14:58:12.410 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-key list | grep Ceph 2022-03-20T14:58:12.546 INFO:teuthology.orchestra.run.smithi134.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-03-20T14:58:12.606 INFO:teuthology.orchestra.run.smithi134.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-03-20T14:58:12.607 INFO:teuthology.orchestra.run.smithi134.stdout:uid [ unknown] Ceph.com (release key) 2022-03-20T14:58:12.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo lsb_release -is 2022-03-20T14:58:12.660 INFO:teuthology.orchestra.run.smithi134.stdout:Ubuntu 2022-03-20T14:58:12.666 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-20T14:58:12.666 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-03-20T14:58:12.667 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-20T14:58:12.856 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default/ 2022-03-20T14:58:12.857 INFO:teuthology.task.install.deb:Package version is 17.0.0-11187-ge92561bf-1focal 2022-03-20T14:58:12.974 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T14:58:12.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-03-20T14:58:12.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get update 2022-03-20T14:58:13.183 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:58:13.192 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:58:13.211 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:58:13.252 INFO:teuthology.orchestra.run.smithi134.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal InRelease 2022-03-20T14:58:13.276 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-03-20T14:58:13.292 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:58:13.301 INFO:teuthology.orchestra.run.smithi134.stdout:Ign:8 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal Release.gpg 2022-03-20T14:58:13.324 INFO:teuthology.orchestra.run.smithi134.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2022-03-20T14:58:13.356 INFO:teuthology.orchestra.run.smithi134.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main i386 Packages [3,367 B] 2022-03-20T14:58:13.395 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease 2022-03-20T14:58:13.888 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 28.4 kB in 1s (43.2 kB/s) 2022-03-20T14:58:14.826 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:58:14.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-11187-ge92561bf-1focal cephadm=17.0.0-11187-ge92561bf-1focal ceph-mds=17.0.0-11187-ge92561bf-1focal ceph-mgr=17.0.0-11187-ge92561bf-1focal ceph-common=17.0.0-11187-ge92561bf-1focal ceph-fuse=17.0.0-11187-ge92561bf-1focal ceph-test=17.0.0-11187-ge92561bf-1focal ceph-volume=17.0.0-11187-ge92561bf-1focal radosgw=17.0.0-11187-ge92561bf-1focal python3-rados=17.0.0-11187-ge92561bf-1focal python3-rgw=17.0.0-11187-ge92561bf-1focal python3-cephfs=17.0.0-11187-ge92561bf-1focal python3-rbd=17.0.0-11187-ge92561bf-1focal libcephfs2=17.0.0-11187-ge92561bf-1focal libcephfs-dev=17.0.0-11187-ge92561bf-1focal librados2=17.0.0-11187-ge92561bf-1focal librbd1=17.0.0-11187-ge92561bf-1focal rbd-fuse=17.0.0-11187-ge92561bf-1focal 2022-03-20T14:58:14.912 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2022-03-20T14:58:15.098 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2022-03-20T14:58:15.099 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2022-03-20T14:58:15.259 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages were automatically installed and are no longer required: 2022-03-20T14:58:15.260 INFO:teuthology.orchestra.run.smithi134.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-03-20T14:58:15.260 INFO:teuthology.orchestra.run.smithi134.stdout:Use 'sudo apt autoremove' to remove them. 2022-03-20T14:58:15.261 INFO:teuthology.orchestra.run.smithi134.stdout:The following additional packages will be installed: 2022-03-20T14:58:15.261 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-03-20T14:58:15.262 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjq1 2022-03-20T14:58:15.262 INFO:teuthology.orchestra.run.smithi134.stdout: libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-20T14:58:15.263 INFO:teuthology.orchestra.run.smithi134.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-20T14:58:15.263 INFO:teuthology.orchestra.run.smithi134.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-20T14:58:15.263 INFO:teuthology.orchestra.run.smithi134.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-20T14:58:15.263 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-03-20T14:58:15.263 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-20T14:58:15.264 INFO:teuthology.orchestra.run.smithi134.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-20T14:58:15.264 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-20T14:58:15.264 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-20T14:58:15.264 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-20T14:58:15.265 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-20T14:58:15.265 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-03-20T14:58:15.265 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-03-20T14:58:15.265 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-03-20T14:58:15.266 INFO:teuthology.orchestra.run.smithi134.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-03-20T14:58:15.266 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-03-20T14:58:15.266 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket python3-webtest python3-werkzeug xmlstarlet 2022-03-20T14:58:15.266 INFO:teuthology.orchestra.run.smithi134.stdout:Suggested packages: 2022-03-20T14:58:15.266 INFO:teuthology.orchestra.run.smithi134.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-03-20T14:58:15.267 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-vectors python3-genshi python3-crypto python3-lxml-dbg 2022-03-20T14:58:15.267 INFO:teuthology.orchestra.run.smithi134.stdout: python-lxml-doc python3-beaker python-mako-doc python-openssl-doc 2022-03-20T14:58:15.267 INFO:teuthology.orchestra.run.smithi134.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2022-03-20T14:58:15.267 INFO:teuthology.orchestra.run.smithi134.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2022-03-20T14:58:15.268 INFO:teuthology.orchestra.run.smithi134.stdout: python-pyinotify-doc python-scipy-doc python3-dap python-sklearn-doc 2022-03-20T14:58:15.268 INFO:teuthology.orchestra.run.smithi134.stdout: ipython3 python-waitress-doc python-webob-doc python-webtest-doc 2022-03-20T14:58:15.268 INFO:teuthology.orchestra.run.smithi134.stdout: python-werkzeug-doc python3-termcolor python3-watchdog 2022-03-20T14:58:15.268 INFO:teuthology.orchestra.run.smithi134.stdout:Recommended packages: 2022-03-20T14:58:15.268 INFO:teuthology.orchestra.run.smithi134.stdout: btrfs-tools ceph-mgr-cephadm 2022-03-20T14:58:15.373 INFO:teuthology.orchestra.run.smithi134.stdout:The following NEW packages will be installed: 2022-03-20T14:58:15.374 INFO:teuthology.orchestra.run.smithi134.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-dashboard 2022-03-20T14:58:15.374 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-03-20T14:58:15.376 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon ceph-osd ceph-test ceph-volume cephadm jq libcephfs-dev libcephfs2 2022-03-20T14:58:15.376 INFO:teuthology.orchestra.run.smithi134.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 2022-03-20T14:58:15.377 INFO:teuthology.orchestra.run.smithi134.stdout: liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-03-20T14:58:15.377 INFO:teuthology.orchestra.run.smithi134.stdout: libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-03-20T14:58:15.377 INFO:teuthology.orchestra.run.smithi134.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-03-20T14:58:15.377 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2022-03-20T14:58:15.378 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2022-03-20T14:58:15.378 INFO:teuthology.orchestra.run.smithi134.stdout: python3-html5lib python3-joblib python3-jwt python3-kubernetes 2022-03-20T14:58:15.378 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-03-20T14:58:15.378 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib python3-openssl python3-packaging python3-paste 2022-03-20T14:58:15.379 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pluggy 2022-03-20T14:58:15.379 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2022-03-20T14:58:15.379 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2022-03-20T14:58:15.379 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2022-03-20T14:58:15.379 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-03-20T14:58:15.380 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-03-20T14:58:15.380 INFO:teuthology.orchestra.run.smithi134.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-03-20T14:58:15.380 INFO:teuthology.orchestra.run.smithi134.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-03-20T14:58:15.380 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse xmlstarlet 2022-03-20T14:58:15.381 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages will be upgraded: 2022-03-20T14:58:15.381 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 librbd1 2022-03-20T14:58:15.531 INFO:teuthology.orchestra.run.smithi134.stdout:2 upgraded, 95 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T14:58:15.532 INFO:teuthology.orchestra.run.smithi134.stdout:Need to get 181 MB of archives. 2022-03-20T14:58:15.532 INFO:teuthology.orchestra.run.smithi134.stdout:After this operation, 846 MB of additional disk space will be used. 2022-03-20T14:58:15.532 INFO:teuthology.orchestra.run.smithi134.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-11187-ge92561bf-1focal [3,478 kB] 2022-03-20T14:58:15.562 INFO:teuthology.orchestra.run.smithi134.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-03-20T14:58:15.867 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-11187-ge92561bf-1focal [3,534 kB] 2022-03-20T14:58:15.924 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-03-20T14:58:15.953 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-03-20T14:58:16.014 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-03-20T14:58:16.052 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-03-20T14:58:16.053 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-11187-ge92561bf-1focal [755 kB] 2022-03-20T14:58:16.058 INFO:teuthology.orchestra.run.smithi134.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-11187-ge92561bf-1focal [412 kB] 2022-03-20T14:58:16.080 INFO:teuthology.orchestra.run.smithi134.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-11187-ge92561bf-1focal [22.4 kB] 2022-03-20T14:58:16.081 INFO:teuthology.orchestra.run.smithi134.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-11187-ge92561bf-1focal [200 kB] 2022-03-20T14:58:16.085 INFO:teuthology.orchestra.run.smithi134.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-11187-ge92561bf-1focal [42.0 kB] 2022-03-20T14:58:16.087 INFO:teuthology.orchestra.run.smithi134.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-11187-ge92561bf-1focal [385 kB] 2022-03-20T14:58:16.101 INFO:teuthology.orchestra.run.smithi134.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-11187-ge92561bf-1focal [4,164 kB] 2022-03-20T14:58:16.124 INFO:teuthology.orchestra.run.smithi134.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-03-20T14:58:16.131 INFO:teuthology.orchestra.run.smithi134.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-03-20T14:58:16.165 INFO:teuthology.orchestra.run.smithi134.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-03-20T14:58:16.169 INFO:teuthology.orchestra.run.smithi134.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-03-20T14:58:16.173 INFO:teuthology.orchestra.run.smithi134.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-03-20T14:58:16.189 INFO:teuthology.orchestra.run.smithi134.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-03-20T14:58:16.192 INFO:teuthology.orchestra.run.smithi134.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-03-20T14:58:16.193 INFO:teuthology.orchestra.run.smithi134.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-03-20T14:58:16.194 INFO:teuthology.orchestra.run.smithi134.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-03-20T14:58:16.246 INFO:teuthology.orchestra.run.smithi134.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-03-20T14:58:16.247 INFO:teuthology.orchestra.run.smithi134.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-03-20T14:58:16.248 INFO:teuthology.orchestra.run.smithi134.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-03-20T14:58:16.255 INFO:teuthology.orchestra.run.smithi134.stdout:Get:27 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-11187-ge92561bf-1focal [116 kB] 2022-03-20T14:58:16.258 INFO:teuthology.orchestra.run.smithi134.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-11187-ge92561bf-1focal [477 kB] 2022-03-20T14:58:16.266 INFO:teuthology.orchestra.run.smithi134.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-11187-ge92561bf-1focal [24.4 MB] 2022-03-20T14:58:16.326 INFO:teuthology.orchestra.run.smithi134.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-03-20T14:58:16.327 INFO:teuthology.orchestra.run.smithi134.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-03-20T14:58:16.331 INFO:teuthology.orchestra.run.smithi134.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2022-03-20T14:58:16.333 INFO:teuthology.orchestra.run.smithi134.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-03-20T14:58:16.335 INFO:teuthology.orchestra.run.smithi134.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-03-20T14:58:16.349 INFO:teuthology.orchestra.run.smithi134.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-03-20T14:58:16.349 INFO:teuthology.orchestra.run.smithi134.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-03-20T14:58:16.406 INFO:teuthology.orchestra.run.smithi134.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-03-20T14:58:16.407 INFO:teuthology.orchestra.run.smithi134.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-03-20T14:58:16.411 INFO:teuthology.orchestra.run.smithi134.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-03-20T14:58:16.486 INFO:teuthology.orchestra.run.smithi134.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-03-20T14:58:16.487 INFO:teuthology.orchestra.run.smithi134.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-03-20T14:58:16.490 INFO:teuthology.orchestra.run.smithi134.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-03-20T14:58:16.493 INFO:teuthology.orchestra.run.smithi134.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-03-20T14:58:16.495 INFO:teuthology.orchestra.run.smithi134.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-03-20T14:58:16.495 INFO:teuthology.orchestra.run.smithi134.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-03-20T14:58:16.496 INFO:teuthology.orchestra.run.smithi134.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-03-20T14:58:16.826 INFO:teuthology.orchestra.run.smithi134.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-03-20T14:58:16.863 INFO:teuthology.orchestra.run.smithi134.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-03-20T14:58:16.885 INFO:teuthology.orchestra.run.smithi134.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-03-20T14:58:16.922 INFO:teuthology.orchestra.run.smithi134.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-03-20T14:58:16.922 INFO:teuthology.orchestra.run.smithi134.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-03-20T14:58:16.923 INFO:teuthology.orchestra.run.smithi134.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-03-20T14:58:16.923 INFO:teuthology.orchestra.run.smithi134.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-03-20T14:58:16.924 INFO:teuthology.orchestra.run.smithi134.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-03-20T14:58:16.924 INFO:teuthology.orchestra.run.smithi134.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-03-20T14:58:16.925 INFO:teuthology.orchestra.run.smithi134.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-03-20T14:58:16.987 INFO:teuthology.orchestra.run.smithi134.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-03-20T14:58:16.988 INFO:teuthology.orchestra.run.smithi134.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-03-20T14:58:17.067 INFO:teuthology.orchestra.run.smithi134.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-03-20T14:58:17.145 INFO:teuthology.orchestra.run.smithi134.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-03-20T14:58:17.146 INFO:teuthology.orchestra.run.smithi134.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-11187-ge92561bf-1focal [6,427 kB] 2022-03-20T14:58:17.202 INFO:teuthology.orchestra.run.smithi134.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-03-20T14:58:17.203 INFO:teuthology.orchestra.run.smithi134.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-03-20T14:58:17.204 INFO:teuthology.orchestra.run.smithi134.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-03-20T14:58:17.205 INFO:teuthology.orchestra.run.smithi134.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-03-20T14:58:17.207 INFO:teuthology.orchestra.run.smithi134.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-03-20T14:58:17.210 INFO:teuthology.orchestra.run.smithi134.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-03-20T14:58:17.211 INFO:teuthology.orchestra.run.smithi134.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-03-20T14:58:17.211 INFO:teuthology.orchestra.run.smithi134.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-03-20T14:58:17.232 INFO:teuthology.orchestra.run.smithi134.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-03-20T14:58:17.293 INFO:teuthology.orchestra.run.smithi134.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-03-20T14:58:17.306 INFO:teuthology.orchestra.run.smithi134.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-03-20T14:58:17.307 INFO:teuthology.orchestra.run.smithi134.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-03-20T14:58:17.308 INFO:teuthology.orchestra.run.smithi134.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-03-20T14:58:17.308 INFO:teuthology.orchestra.run.smithi134.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-03-20T14:58:17.310 INFO:teuthology.orchestra.run.smithi134.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-03-20T14:58:17.310 INFO:teuthology.orchestra.run.smithi134.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-03-20T14:58:17.311 INFO:teuthology.orchestra.run.smithi134.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-03-20T14:58:17.311 INFO:teuthology.orchestra.run.smithi134.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-03-20T14:58:17.355 INFO:teuthology.orchestra.run.smithi134.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-11187-ge92561bf-1focal [211 kB] 2022-03-20T14:58:17.356 INFO:teuthology.orchestra.run.smithi134.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-11187-ge92561bf-1focal [151 kB] 2022-03-20T14:58:17.358 INFO:teuthology.orchestra.run.smithi134.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-11187-ge92561bf-1focal [1,574 kB] 2022-03-20T14:58:17.370 INFO:teuthology.orchestra.run.smithi134.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-11187-ge92561bf-1focal [7,818 kB] 2022-03-20T14:58:17.388 INFO:teuthology.orchestra.run.smithi134.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-03-20T14:58:17.611 INFO:teuthology.orchestra.run.smithi134.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-11187-ge92561bf-1focal [28.6 MB] 2022-03-20T14:58:18.651 INFO:teuthology.orchestra.run.smithi134.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-11187-ge92561bf-1focal [3,816 B] 2022-03-20T14:58:18.652 INFO:teuthology.orchestra.run.smithi134.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-11187-ge92561bf-1focal [863 kB] 2022-03-20T14:58:18.658 INFO:teuthology.orchestra.run.smithi134.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-11187-ge92561bf-1focal [2,397 kB] 2022-03-20T14:58:18.677 INFO:teuthology.orchestra.run.smithi134.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-11187-ge92561bf-1focal [1,678 kB] 2022-03-20T14:58:18.710 INFO:teuthology.orchestra.run.smithi134.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-11187-ge92561bf-1focal [7,890 kB] 2022-03-20T14:58:18.970 INFO:teuthology.orchestra.run.smithi134.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-11187-ge92561bf-1focal [3,928 B] 2022-03-20T14:58:18.970 INFO:teuthology.orchestra.run.smithi134.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-11187-ge92561bf-1focal [50.5 MB] 2022-03-20T14:58:20.956 INFO:teuthology.orchestra.run.smithi134.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-11187-ge92561bf-1focal [219 kB] 2022-03-20T14:58:20.997 INFO:teuthology.orchestra.run.smithi134.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-11187-ge92561bf-1focal [69.7 kB] 2022-03-20T14:58:20.998 INFO:teuthology.orchestra.run.smithi134.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-11187-ge92561bf-1focal [21.7 kB] 2022-03-20T14:58:20.999 INFO:teuthology.orchestra.run.smithi134.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-11187-ge92561bf-1focal [12.7 MB] 2022-03-20T14:58:21.401 INFO:teuthology.orchestra.run.smithi134.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-yuri6-testing-2022-03-18-1104/e92561bf8a1bc96ca490936c6f46ac4673e1b26e/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-11187-ge92561bf-1focal [78.8 kB] 2022-03-20T14:58:22.192 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 181 MB in 6s (29.8 MB/s) 2022-03-20T14:58:22.746 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-03-20T14:58:22.795 INFO:teuthology.orchestra.run.smithi134.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109337 files and directories currently installed.) 2022-03-20T14:58:22.799 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../00-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-03-20T14:58:22.905 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-20T14:58:23.337 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../01-librbd1_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:23.474 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librbd1 (17.0.0-11187-ge92561bf-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-20T14:58:24.183 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../02-librados2_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:24.304 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librados2 (17.0.0-11187-ge92561bf-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-03-20T14:58:25.131 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libcephfs2. 2022-03-20T14:58:25.146 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../03-libcephfs2_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:25.205 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libcephfs2 (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:25.563 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rados. 2022-03-20T14:58:25.576 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../04-python3-rados_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:25.606 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rados (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:26.195 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-03-20T14:58:26.208 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../05-python3-ceph-argparse_17.0.0-11187-ge92561bf-1focal_all.deb ... 2022-03-20T14:58:26.254 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-ceph-argparse (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:26.496 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cephfs. 2022-03-20T14:58:26.508 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../06-python3-cephfs_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:26.547 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cephfs (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:26.934 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-ceph-common. 2022-03-20T14:58:26.948 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../07-python3-ceph-common_17.0.0-11187-ge92561bf-1focal_all.deb ... 2022-03-20T14:58:26.991 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-ceph-common (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:27.413 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-prettytable. 2022-03-20T14:58:27.427 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../08-python3-prettytable_0.7.2-5_all.deb ... 2022-03-20T14:58:27.473 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-03-20T14:58:27.756 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rbd. 2022-03-20T14:58:27.770 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../09-python3-rbd_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:27.838 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rbd (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:28.190 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-03-20T14:58:28.203 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../10-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-03-20T14:58:28.252 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-20T14:58:28.720 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-03-20T14:58:28.729 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../11-librabbitmq4_0.10.0-1_amd64.deb ... 2022-03-20T14:58:28.774 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-03-20T14:58:29.132 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-03-20T14:58:29.144 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../12-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-03-20T14:58:29.190 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-20T14:58:29.653 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librgw2. 2022-03-20T14:58:29.666 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../13-librgw2_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:29.711 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librgw2 (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:30.372 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rgw. 2022-03-20T14:58:30.384 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../14-python3-rgw_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:30.422 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rgw (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:30.936 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liboath0. 2022-03-20T14:58:30.952 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../15-liboath0_2.6.1-1.3_amd64.deb ... 2022-03-20T14:58:30.996 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-03-20T14:58:31.441 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libradosstriper1. 2022-03-20T14:58:31.450 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../16-libradosstriper1_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:31.491 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libradosstriper1 (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:32.007 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-common. 2022-03-20T14:58:32.022 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../17-ceph-common_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:32.065 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-common (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:34.700 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-base. 2022-03-20T14:58:34.714 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../18-ceph-base_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:34.765 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-base (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:35.883 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cryptography. 2022-03-20T14:58:35.899 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../19-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-03-20T14:58:35.944 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-20T14:58:36.546 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-openssl. 2022-03-20T14:58:36.558 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../20-python3-openssl_19.0.0-1build1_all.deb ... 2022-03-20T14:58:36.595 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-03-20T14:58:36.874 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-03-20T14:58:36.883 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../21-ceph-mgr-modules-core_17.0.0-11187-ge92561bf-1focal_all.deb ... 2022-03-20T14:58:36.942 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-modules-core (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:37.404 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-bcrypt. 2022-03-20T14:58:37.418 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../22-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-03-20T14:58:37.463 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-20T14:58:37.845 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cherrypy3. 2022-03-20T14:58:37.857 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../23-python3-cherrypy3_8.9.1-6_all.deb ... 2022-03-20T14:58:37.894 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-03-20T14:58:38.197 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jwt. 2022-03-20T14:58:38.210 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../24-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-03-20T14:58:38.246 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-03-20T14:58:38.657 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-logutils. 2022-03-20T14:58:38.668 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../25-python3-logutils_0.3.3-7_all.deb ... 2022-03-20T14:58:38.703 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-03-20T14:58:38.982 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-markupsafe. 2022-03-20T14:58:38.990 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../26-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-03-20T14:58:39.031 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-03-20T14:58:39.361 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-mako. 2022-03-20T14:58:39.369 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../27-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-03-20T14:58:39.411 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-20T14:58:39.843 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-simplegeneric. 2022-03-20T14:58:39.855 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../28-python3-simplegeneric_0.8.1-3_all.deb ... 2022-03-20T14:58:39.893 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-03-20T14:58:40.223 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-singledispatch. 2022-03-20T14:58:40.238 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../29-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-03-20T14:58:40.279 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-03-20T14:58:40.625 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-webob. 2022-03-20T14:58:40.642 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../30-python3-webob_1%3a1.8.5-2_all.deb ... 2022-03-20T14:58:40.704 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-03-20T14:58:41.082 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-soupsieve. 2022-03-20T14:58:41.089 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../31-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-03-20T14:58:41.134 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-20T14:58:41.597 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-bs4. 2022-03-20T14:58:41.609 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../32-python3-bs4_4.8.2-1_all.deb ... 2022-03-20T14:58:41.651 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-03-20T14:58:42.121 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-waitress. 2022-03-20T14:58:42.134 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../33-python3-waitress_1.4.1-1_all.deb ... 2022-03-20T14:58:42.191 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-waitress (1.4.1-1) ... 2022-03-20T14:58:42.645 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-tempita. 2022-03-20T14:58:42.658 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../34-python3-tempita_0.5.2-6_all.deb ... 2022-03-20T14:58:42.703 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-03-20T14:58:43.089 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-paste. 2022-03-20T14:58:43.102 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../35-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-03-20T14:58:43.139 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-03-20T14:58:43.576 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-03-20T14:58:43.590 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../36-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-03-20T14:58:43.658 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-03-20T14:58:44.083 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pastedeploy. 2022-03-20T14:58:44.097 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../37-python3-pastedeploy_2.1.0-1_all.deb ... 2022-03-20T14:58:44.136 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-03-20T14:58:44.463 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-webtest. 2022-03-20T14:58:44.475 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../38-python3-webtest_2.0.32-2_all.deb ... 2022-03-20T14:58:44.530 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-03-20T14:58:44.983 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pecan. 2022-03-20T14:58:44.997 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../39-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-03-20T14:58:45.041 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-03-20T14:58:45.422 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-werkzeug. 2022-03-20T14:58:45.436 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../40-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-03-20T14:58:45.466 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-20T14:58:45.836 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-03-20T14:58:45.848 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../41-libsqlite3-mod-ceph_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:45.915 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:46.481 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr. 2022-03-20T14:58:46.493 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../42-ceph-mgr_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:46.531 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:47.076 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mon. 2022-03-20T14:58:47.091 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../43-ceph-mon_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:47.142 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mon (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:48.396 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-osd. 2022-03-20T14:58:48.408 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../44-ceph-osd_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:48.447 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-osd (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:51.419 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph. 2022-03-20T14:58:51.431 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../45-ceph_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:51.540 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:51.918 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-fuse. 2022-03-20T14:58:51.932 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../46-ceph-fuse_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:51.993 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-fuse (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:52.454 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mds. 2022-03-20T14:58:52.464 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../47-ceph-mds_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:58:52.570 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mds (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:53.890 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-repoze.lru. 2022-03-20T14:58:53.905 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2022-03-20T14:58:53.974 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-03-20T14:58:54.281 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libjs-underscore. 2022-03-20T14:58:54.294 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../49-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-03-20T14:58:54.343 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-20T14:58:54.633 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-03-20T14:58:54.644 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../50-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-03-20T14:58:54.691 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-20T14:58:55.090 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-routes. 2022-03-20T14:58:55.099 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../51-python3-routes_2.4.1-2_all.deb ... 2022-03-20T14:58:55.163 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-03-20T14:58:55.549 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-03-20T14:58:55.562 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../52-ceph-mgr-dashboard_17.0.0-11187-ge92561bf-1focal_all.deb ... 2022-03-20T14:58:55.607 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-dashboard (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:58:56.241 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-decorator. 2022-03-20T14:58:56.254 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../53-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-03-20T14:58:56.324 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-03-20T14:58:56.692 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-03-20T14:58:56.702 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../54-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-03-20T14:58:56.742 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-20T14:58:57.110 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-scipy. 2022-03-20T14:58:57.125 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../55-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-03-20T14:58:57.168 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-03-20T14:58:58.931 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-03-20T14:58:58.945 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../56-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-03-20T14:58:58.989 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-20T14:58:59.695 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-joblib. 2022-03-20T14:58:59.710 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../57-python3-joblib_0.14.0-3_all.deb ... 2022-03-20T14:58:59.751 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-03-20T14:59:00.143 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-sklearn. 2022-03-20T14:59:00.152 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../58-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-03-20T14:59:00.201 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-20T14:59:01.264 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-03-20T14:59:01.280 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../59-ceph-mgr-diskprediction-local_17.0.0-11187-ge92561bf-1focal_all.deb ... 2022-03-20T14:59:01.350 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:02.906 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cachetools. 2022-03-20T14:59:02.919 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../60-python3-cachetools_4.0.0-1_all.deb ... 2022-03-20T14:59:02.964 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-03-20T14:59:03.460 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pyasn1. 2022-03-20T14:59:03.473 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../61-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-03-20T14:59:03.511 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-03-20T14:59:03.859 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-03-20T14:59:03.873 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../62-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-03-20T14:59:03.917 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-20T14:59:04.405 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rsa. 2022-03-20T14:59:04.419 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../63-python3-rsa_4.0-3_all.deb ... 2022-03-20T14:59:04.477 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rsa (4.0-3) ... 2022-03-20T14:59:04.861 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-google-auth. 2022-03-20T14:59:04.869 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../64-python3-google-auth_1.5.1-2_all.deb ... 2022-03-20T14:59:04.911 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-03-20T14:59:05.239 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-blinker. 2022-03-20T14:59:05.252 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../65-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-03-20T14:59:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-20T14:59:05.678 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-oauthlib. 2022-03-20T14:59:05.687 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../66-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-03-20T14:59:05.734 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-20T14:59:06.100 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-03-20T14:59:06.111 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../67-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-03-20T14:59:06.158 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-20T14:59:06.642 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-websocket. 2022-03-20T14:59:06.652 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../68-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-03-20T14:59:06.703 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-03-20T14:59:07.194 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-kubernetes. 2022-03-20T14:59:07.207 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../69-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-03-20T14:59:07.243 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-03-20T14:59:07.924 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-03-20T14:59:07.938 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../70-ceph-mgr-k8sevents_17.0.0-11187-ge92561bf-1focal_all.deb ... 2022-03-20T14:59:07.991 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:08.340 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libonig5:amd64. 2022-03-20T14:59:08.353 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../71-libonig5_6.9.4-1_amd64.deb ... 2022-03-20T14:59:08.406 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-03-20T14:59:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libjq1:amd64. 2022-03-20T14:59:08.975 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../72-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-20T14:59:09.019 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-20T14:59:09.321 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package jq. 2022-03-20T14:59:09.332 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../73-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-03-20T14:59:09.371 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-03-20T14:59:09.929 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-03-20T14:59:09.943 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../74-libxslt1.1_1.1.34-4_amd64.deb ... 2022-03-20T14:59:10.004 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-03-20T14:59:10.283 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package xmlstarlet. 2022-03-20T14:59:10.296 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../75-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-03-20T14:59:10.340 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-03-20T14:59:10.749 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-test. 2022-03-20T14:59:10.764 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../76-ceph-test_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:59:10.807 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-test (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:15.906 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-volume. 2022-03-20T14:59:15.919 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../77-ceph-volume_17.0.0-11187-ge92561bf-1focal_all.deb ... 2022-03-20T14:59:15.998 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-volume (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:16.441 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package cephadm. 2022-03-20T14:59:16.455 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../78-cephadm_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:59:16.516 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking cephadm (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:16.814 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libcephfs-dev. 2022-03-20T14:59:16.825 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../79-libcephfs-dev_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:59:16.911 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libcephfs-dev (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:17.515 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package nvme-cli. 2022-03-20T14:59:17.529 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../80-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-03-20T14:59:17.608 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-03-20T14:59:18.105 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-atomicwrites. 2022-03-20T14:59:18.118 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../81-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-03-20T14:59:18.211 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-03-20T14:59:18.667 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-attr. 2022-03-20T14:59:18.682 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../82-python3-attr_19.3.0-2_all.deb ... 2022-03-20T14:59:18.721 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-03-20T14:59:19.110 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-webencodings. 2022-03-20T14:59:19.123 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../83-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-03-20T14:59:19.167 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-20T14:59:19.596 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-html5lib. 2022-03-20T14:59:19.608 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../84-python3-html5lib_1.0.1-2_all.deb ... 2022-03-20T14:59:19.650 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-03-20T14:59:20.013 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-03-20T14:59:20.029 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../85-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-03-20T14:59:20.071 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-20T14:59:20.714 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-packaging. 2022-03-20T14:59:20.725 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../86-python3-packaging_20.3-1_all.deb ... 2022-03-20T14:59:20.763 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-packaging (20.3-1) ... 2022-03-20T14:59:21.159 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pastescript. 2022-03-20T14:59:21.172 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../87-python3-pastescript_2.0.2-4_all.deb ... 2022-03-20T14:59:21.217 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-03-20T14:59:21.660 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pluggy. 2022-03-20T14:59:21.673 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../88-python3-pluggy_0.13.0-2_all.deb ... 2022-03-20T14:59:21.718 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-03-20T14:59:22.089 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-psutil. 2022-03-20T14:59:22.105 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../89-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-03-20T14:59:22.147 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-03-20T14:59:22.630 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-py. 2022-03-20T14:59:22.643 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../90-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-03-20T14:59:22.697 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-03-20T14:59:23.043 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pyinotify. 2022-03-20T14:59:23.055 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../91-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-03-20T14:59:23.093 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-20T14:59:23.487 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-wcwidth. 2022-03-20T14:59:23.502 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../92-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-03-20T14:59:23.549 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-20T14:59:24.047 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pytest. 2022-03-20T14:59:24.059 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../93-python3-pytest_4.6.9-1_all.deb ... 2022-03-20T14:59:24.096 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-03-20T14:59:24.484 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-simplejson. 2022-03-20T14:59:24.498 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../94-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-03-20T14:59:24.865 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-20T14:59:26.067 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package radosgw. 2022-03-20T14:59:26.076 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../95-radosgw_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:59:26.129 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking radosgw (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:27.613 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package rbd-fuse. 2022-03-20T14:59:27.622 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../96-rbd-fuse_17.0.0-11187-ge92561bf-1focal_amd64.deb ... 2022-03-20T14:59:27.730 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking rbd-fuse (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:28.040 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-attr (19.3.0-2) ... 2022-03-20T14:59:28.382 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-03-20T14:59:28.550 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-03-20T14:59:28.821 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up cephadm (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:29.526 INFO:teuthology.orchestra.run.smithi134.stdout:Adding system user cephadm....done 2022-03-20T14:59:29.732 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-waitress (1.4.1-1) ... 2022-03-20T14:59:30.124 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-03-20T14:59:30.461 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-03-20T14:59:30.596 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-03-20T14:59:30.947 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-03-20T14:59:31.374 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-03-20T14:59:31.707 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-03-20T14:59:32.043 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-03-20T14:59:32.385 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-03-20T14:59:32.695 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-ceph-argparse (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:33.022 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-03-20T14:59:33.173 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-03-20T14:59:33.564 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-03-20T14:59:33.859 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-03-20T14:59:34.244 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-03-20T14:59:34.510 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-03-20T14:59:34.879 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-03-20T14:59:35.180 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-03-20T14:59:35.505 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-packaging (20.3-1) ... 2022-03-20T14:59:35.807 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-03-20T14:59:35.958 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-03-20T14:59:36.267 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-03-20T14:59:36.705 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-03-20T14:59:37.071 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-03-20T14:59:37.473 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-ceph-common (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:37.864 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-03-20T14:59:38.224 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-03-20T14:59:38.376 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-03-20T14:59:38.718 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-03-20T14:59:38.911 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-03-20T14:59:39.069 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-03-20T14:59:39.413 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-03-20T14:59:39.706 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-03-20T14:59:40.073 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-03-20T14:59:40.241 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-03-20T14:59:40.526 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-03-20T14:59:40.867 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-03-20T14:59:41.202 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-03-20T14:59:41.300 INFO:teuthology.orchestra.run.smithi134.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-03-20T14:59:41.556 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-03-20T14:59:41.703 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-03-20T14:59:42.100 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-03-20T14:59:42.582 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-03-20T14:59:42.742 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-03-20T14:59:42.946 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-03-20T14:59:43.318 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-03-20T14:59:43.687 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librados2 (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:43.819 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-03-20T14:59:45.490 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librgw2 (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:45.611 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libsqlite3-mod-ceph (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:45.788 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-03-20T14:59:45.929 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-03-20T14:59:46.373 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-03-20T14:59:46.656 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-03-20T14:59:46.966 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-03-20T14:59:47.309 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libcephfs2 (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:47.433 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-03-20T14:59:47.796 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libradosstriper1 (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:47.956 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-03-20T14:59:48.297 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-03-20T14:59:49.444 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librbd1 (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:49.587 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-03-20T14:59:49.797 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-03-20T14:59:49.954 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-03-20T14:59:50.264 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-03-20T14:59:50.391 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rsa (4.0-3) ... 2022-03-20T14:59:50.684 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-03-20T14:59:51.026 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-modules-core (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:51.159 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-fuse (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:51.376 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-20T14:59:51.377 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-03-20T14:59:52.337 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libcephfs-dev (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:52.471 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-03-20T14:59:52.786 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-03-20T14:59:53.083 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-03-20T14:59:53.468 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rados (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:53.646 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rbd (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:53.778 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up rbd-fuse (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:53.954 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-routes (2.4.1-2) ... 2022-03-20T14:59:54.298 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-03-20T14:59:54.699 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rgw (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:54.848 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cephfs (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:54.974 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-03-20T14:59:56.468 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-common (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T14:59:57.596 INFO:teuthology.orchestra.run.smithi134.stdout:Adding group ceph....done 2022-03-20T14:59:58.058 INFO:teuthology.orchestra.run.smithi134.stdout:Adding system user ceph....done 2022-03-20T14:59:58.260 INFO:teuthology.orchestra.run.smithi134.stdout:Setting system user ceph properties....done 2022-03-20T14:59:58.297 INFO:teuthology.orchestra.run.smithi134.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-03-20T14:59:58.452 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-03-20T14:59:58.977 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-03-20T14:59:59.914 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-test (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:00.077 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up radosgw (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:00.366 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-20T15:00:00.367 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-03-20T15:00:01.697 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-base (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:02.133 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-03-20T15:00:03.113 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mds (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-20T15:00:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-03-20T15:00:04.247 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:04.476 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-20T15:00:04.477 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-03-20T15:00:05.429 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-osd (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:06.144 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-20T15:00:06.145 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-03-20T15:00:07.124 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-k8sevents (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:07.431 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:07.557 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mon (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:07.784 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-20T15:00:07.785 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-03-20T15:00:08.729 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:08.850 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-dashboard (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:09.010 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-volume (17.0.0-11187-ge92561bf-1focal) ... 2022-03-20T15:00:09.165 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for systemd (245.4-4ubuntu3.15) ... 2022-03-20T15:00:09.725 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T15:00:12.585 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-20T15:00:13.104 INFO:teuthology.orchestra.run.smithi134.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-03-20T15:00:13.107 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:13.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo lsb_release -is 2022-03-20T15:00:13.207 INFO:teuthology.orchestra.run.smithi134.stdout:Ubuntu 2022-03-20T15:00:13.212 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-20T15:00:13.213 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-20T15:00:13.374 DEBUG:teuthology.orchestra.run.smithi134:> dpkg-query -W -f '${Version}' ceph 2022-03-20T15:00:13.399 INFO:teuthology.orchestra.run.smithi134.stdout:17.0.0-11187-ge92561bf-1focal 2022-03-20T15:00:13.400 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11187-ge92561bf-1focal 2022-03-20T15:00:13.401 INFO:teuthology.task.install:The correct ceph version 17.0.0-11187-ge92561bf-1focal is installed. 2022-03-20T15:00:13.402 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-03-20T15:00:13.402 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T15:00:13.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-03-20T15:00:13.418 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-03-20T15:00:13.419 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T15:00:13.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2022-03-20T15:00:13.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-03-20T15:00:13.549 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-03-20T15:00:13.549 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2022-03-20T15:00:13.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2022-03-20T15:00:13.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-03-20T15:00:13.675 INFO:teuthology.run_tasks:Running task radosbench... 2022-03-20T15:00:13.696 INFO:tasks.radosbench:Beginning radosbench... 2022-03-20T15:00:13.697 INFO:tasks.rook.ceph_manager.ceph:creating pool_name unique_pool_0 2022-03-20T15:00:13.697 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd pool create unique_pool_0 16 2022-03-20T15:00:15.020 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_0' created 2022-03-20T15:00:15.040 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2022-03-20T15:00:16.078 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rados' on pool 'unique_pool_0' 2022-03-20T15:00:17.105 DEBUG:teuthology.orchestra.run.smithi134:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.a -b 65536 --object-size 65536 -p unique_pool_0 bench 360 write' 2022-03-20T15:00:17.108 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-20T15:00:17.121 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch host label add '`hostname`' 'foo 2022-03-20T15:00:17.121 DEBUG:teuthology.orchestra.run.smithi134:> ceph' orch host ls '|' grep 'foo 2022-03-20T15:00:17.122 DEBUG:teuthology.orchestra.run.smithi134:> ceph' orch host label rm '`hostname`' 'foo 2022-03-20T15:00:17.122 DEBUG:teuthology.orchestra.run.smithi134:> ceph' orch host ls '|' grep -v 'foo 2022-03-20T15:00:17.122 DEBUG:teuthology.orchestra.run.smithi134:> ' 2022-03-20T15:00:17.170 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:hints = 1 2022-03-20T15:00:17.170 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 360 seconds or 0 objects 2022-03-20T15:00:17.170 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Object prefix: benchmark_data_smithi134_46441 2022-03-20T15:00:17.171 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:00:17.171 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 0 0 0 0 0 0 - 0 2022-03-20T15:00:17.550 INFO:teuthology.orchestra.run.smithi134.stdout:ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-03-20T15:00:17.586 INFO:teuthology.run_tasks:Running task exec... 2022-03-20T15:00:17.599 INFO:teuthology.task.exec:Executing custom commands... 2022-03-20T15:00:17.600 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi134.front.sepia.ceph.com 2022-03-20T15:00:17.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-20T15:00:17.601 DEBUG:teuthology.orchestra.run.smithi134:> toolbox() { 2022-03-20T15:00:17.601 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-20T15:00:17.601 DEBUG:teuthology.orchestra.run.smithi134:> } 2022-03-20T15:00:17.602 DEBUG:teuthology.orchestra.run.smithi134:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-20T15:00:17.602 DEBUG:teuthology.orchestra.run.smithi134:> toolbox ceph orch osd rm 0 --force 2022-03-20T15:00:17.602 DEBUG:teuthology.orchestra.run.smithi134:> removed_pv="" 2022-03-20T15:00:17.603 DEBUG:teuthology.orchestra.run.smithi134:> while [ "$removed_pv" = "" ] 2022-03-20T15:00:17.603 DEBUG:teuthology.orchestra.run.smithi134:> do 2022-03-20T15:00:17.603 DEBUG:teuthology.orchestra.run.smithi134:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-20T15:00:17.604 DEBUG:teuthology.orchestra.run.smithi134:> sleep 3s 2022-03-20T15:00:17.604 DEBUG:teuthology.orchestra.run.smithi134:> done 2022-03-20T15:00:17.604 DEBUG:teuthology.orchestra.run.smithi134:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-20T15:00:17.605 DEBUG:teuthology.orchestra.run.smithi134:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-20T15:00:17.605 DEBUG:teuthology.orchestra.run.smithi134:> toolbox ceph orch device zap $host $target_path --force 2022-03-20T15:00:17.605 DEBUG:teuthology.orchestra.run.smithi134:> zap_completion="0" 2022-03-20T15:00:17.606 DEBUG:teuthology.orchestra.run.smithi134:> while [ "$zap_completion" = "0" ] 2022-03-20T15:00:17.606 DEBUG:teuthology.orchestra.run.smithi134:> do 2022-03-20T15:00:17.606 DEBUG:teuthology.orchestra.run.smithi134:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-20T15:00:17.607 DEBUG:teuthology.orchestra.run.smithi134:> sleep 3s 2022-03-20T15:00:17.607 DEBUG:teuthology.orchestra.run.smithi134:> done 2022-03-20T15:00:17.607 DEBUG:teuthology.orchestra.run.smithi134:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-20T15:00:17.607 DEBUG:teuthology.orchestra.run.smithi134:> toolbox ceph orch apply osd --all-available-devices 2022-03-20T15:00:17.608 DEBUG:teuthology.orchestra.run.smithi134:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-20T15:00:17.608 DEBUG:teuthology.orchestra.run.smithi134:> num_osd="0" 2022-03-20T15:00:17.608 DEBUG:teuthology.orchestra.run.smithi134:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-20T15:00:17.609 DEBUG:teuthology.orchestra.run.smithi134:> do 2022-03-20T15:00:17.609 DEBUG:teuthology.orchestra.run.smithi134:> echo "waiting for osd to come back up" 2022-03-20T15:00:17.609 DEBUG:teuthology.orchestra.run.smithi134:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-20T15:00:17.609 DEBUG:teuthology.orchestra.run.smithi134:> sleep 30s 2022-03-20T15:00:17.610 DEBUG:teuthology.orchestra.run.smithi134:> done 2022-03-20T15:00:17.610 DEBUG:teuthology.orchestra.run.smithi134:> ' 2022-03-20T15:00:17.625 INFO:teuthology.orchestra.run.smithi134.stderr:++ toolbox ceph osd stat 2022-03-20T15:00:17.625 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:00:17.630 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f3 -d ' ' 2022-03-20T15:00:18.171 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 1 16 1277 1261 78.8081 78.8125 0.00652987 0.012632 2022-03-20T15:00:19.171 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 2 16 2799 2783 86.9621 95.125 0.00544492 0.01147 2022-03-20T15:00:20.157 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:20.172 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 3 16 4250 4234 88.1836 90.6875 0.00979935 0.011292 2022-03-20T15:00:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:+ orig_num_osd=4 2022-03-20T15:00:20.991 INFO:teuthology.orchestra.run.smithi134.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-20T15:00:20.991 INFO:teuthology.orchestra.run.smithi134.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-20T15:00:21.090 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:21.172 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 4 16 5508 5492 85.7925 78.625 0.0152263 0.011639 2022-03-20T15:00:21.993 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:21.992133 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-20T15:00:21.993 INFO:tasks.rook.operator.smithi134.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-20T15:00:21.993 INFO:tasks.rook.operator.smithi134.stdout: Storage: v1.StorageScopeSpec{ 2022-03-20T15:00:21.994 INFO:tasks.rook.operator.smithi134.stdout: ... // 3 identical fields 2022-03-20T15:00:21.994 INFO:tasks.rook.operator.smithi134.stdout: Config: nil, 2022-03-20T15:00:21.994 INFO:tasks.rook.operator.smithi134.stdout: Selection: {}, 2022-03-20T15:00:21.995 INFO:tasks.rook.operator.smithi134.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-20T15:00:21.995 INFO:tasks.rook.operator.smithi134.stdout: {Name: "smithi134-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi134-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:21.995 INFO:tasks.rook.operator.smithi134.stdout: {Name: "smithi134-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi134-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:21.995 INFO:tasks.rook.operator.smithi134.stdout: {Name: "smithi134-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi134-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:21.996 INFO:tasks.rook.operator.smithi134.stdout:- { 2022-03-20T15:00:21.996 INFO:tasks.rook.operator.smithi134.stdout:- Name: "smithi134-lv-4", 2022-03-20T15:00:21.996 INFO:tasks.rook.operator.smithi134.stdout:- Count: 1, 2022-03-20T15:00:21.996 INFO:tasks.rook.operator.smithi134.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-20T15:00:21.996 INFO:tasks.rook.operator.smithi134.stdout:- }, 2022-03-20T15:00:21.997 INFO:tasks.rook.operator.smithi134.stdout: }, 2022-03-20T15:00:21.997 INFO:tasks.rook.operator.smithi134.stdout: }, 2022-03-20T15:00:21.997 INFO:tasks.rook.operator.smithi134.stdout: Annotations: nil, 2022-03-20T15:00:21.998 INFO:tasks.rook.operator.smithi134.stdout: Labels: nil, 2022-03-20T15:00:21.998 INFO:tasks.rook.operator.smithi134.stdout: ... // 20 identical fields 2022-03-20T15:00:21.999 INFO:tasks.rook.operator.smithi134.stdout: } 2022-03-20T15:00:21.999 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:21.992226 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:00:22.027 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:22.026860 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:00:22.148 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:22.148730 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:00:22.172 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 5 16 6713 6697 83.6954 75.3125 0.00574055 0.0119178 2022-03-20T15:00:23.173 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 6 16 6841 6825 71.0805 8 0.0170866 0.0119527 2022-03-20T15:00:24.173 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 7 16 6841 6825 60.9272 0 - 0.0119527 2022-03-20T15:00:25.173 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 8 16 6841 6825 53.3116 0 - 0.0119527 2022-03-20T15:00:26.173 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 9 16 6841 6825 47.3884 0 - 0.0119527 2022-03-20T15:00:27.174 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 10 16 6841 6825 42.6496 0 - 0.0119527 2022-03-20T15:00:28.174 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 11 16 6841 6825 38.7725 0 - 0.0119527 2022-03-20T15:00:28.213 INFO:teuthology.orchestra.run.smithi134.stdout:Success 2022-03-20T15:00:28.243 INFO:teuthology.orchestra.run.smithi134.stderr:+ removed_pv= 2022-03-20T15:00:28.243 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' '' = '' ']' 2022-03-20T15:00:28.244 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl get pv 2022-03-20T15:00:28.244 INFO:teuthology.orchestra.run.smithi134.stderr:++ grep Released 2022-03-20T15:00:28.244 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f1 -d ' ' 2022-03-20T15:00:28.414 INFO:teuthology.orchestra.run.smithi134.stderr:+ removed_pv= 2022-03-20T15:00:28.414 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 3s 2022-03-20T15:00:29.174 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 12 16 6841 6825 35.5416 0 - 0.0119527 2022-03-20T15:00:30.174 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 13 16 6841 6825 32.8078 0 - 0.0119527 2022-03-20T15:00:31.174 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 14 16 6841 6825 30.4644 0 - 0.0119527 2022-03-20T15:00:31.415 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' '' = '' ']' 2022-03-20T15:00:31.416 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl get pv 2022-03-20T15:00:31.416 INFO:teuthology.orchestra.run.smithi134.stderr:++ grep Released 2022-03-20T15:00:31.417 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f1 -d ' ' 2022-03-20T15:00:31.655 INFO:teuthology.orchestra.run.smithi134.stderr:+ removed_pv= 2022-03-20T15:00:31.655 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 3s 2022-03-20T15:00:32.175 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 15 16 6917 6901 28.7502 0.527778 0.0363649 0.0347535 2022-03-20T15:00:33.175 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 16 16 7218 7202 28.129 18.8125 0.00194067 0.0339092 2022-03-20T15:00:34.175 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 17 16 7218 7202 26.4744 0 - 0.0339092 2022-03-20T15:00:34.656 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' '' = '' ']' 2022-03-20T15:00:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl get pv 2022-03-20T15:00:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:++ grep Released 2022-03-20T15:00:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f1 -d ' ' 2022-03-20T15:00:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:+ removed_pv= 2022-03-20T15:00:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 3s 2022-03-20T15:00:35.176 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 18 16 7218 7202 25.0037 0 - 0.0339092 2022-03-20T15:00:35.507 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:35.508674 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-20T15:00:36.175 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 19 16 7218 7202 23.6877 0 - 0.0339092 2022-03-20T15:00:37.175 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:00:37.177563+0000 min lat: 0.00126111 max lat: 9.84108 avg lat: 0.0339092 2022-03-20T15:00:37.176 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:00:37.176 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 20 16 7218 7202 22.5034 0 - 0.0339092 2022-03-20T15:00:38.091 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' '' = '' ']' 2022-03-20T15:00:38.092 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl get pv 2022-03-20T15:00:38.093 INFO:teuthology.orchestra.run.smithi134.stderr:++ grep Released 2022-03-20T15:00:38.093 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f1 -d ' ' 2022-03-20T15:00:38.150 INFO:teuthology.orchestra.run.smithi134.stderr:+ removed_pv=smithi134-lv-4 2022-03-20T15:00:38.151 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 3s 2022-03-20T15:00:38.176 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 21 16 7218 7202 21.4318 0 - 0.0339092 2022-03-20T15:00:38.632 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:38.633580 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:00:38.633 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:38.633604 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:00:38.633 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:38.633619 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11187 quincy", pursuing 2022-03-20T15:00:38.933 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:38.934640 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:00:39.171 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:39.169523 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:00:39.171 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:39.169745 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:00:39.176 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 22 16 7218 7202 20.4577 0 - 0.0339092 2022-03-20T15:00:39.642 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:39.643059 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:00:39.699 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:39.699989 I | op-mon: start running mons 2022-03-20T15:00:39.785 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:39.786406 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:00:39.819 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:39.820691 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T15:00:39.825 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:39.826682 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:00:39.827 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:39.826854 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:00:40.176 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 23 16 7218 7202 19.5682 0 - 0.0339092 2022-03-20T15:00:40.680 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:40.681320 I | op-mon: targeting the mon count 1 2022-03-20T15:00:40.725 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:40.724576 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-20T15:00:40.726 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:40.724700 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:00:41.068 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:41.068722 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:00:41.152 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' smithi134-lv-4 = '' ']' 2022-03-20T15:00:41.152 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl get pv smithi134-lv-4 -o 'jsonpath={.spec.local.path}' 2022-03-20T15:00:41.176 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 24 16 7218 7202 18.7529 0 - 0.0339092 2022-03-20T15:00:41.207 INFO:teuthology.orchestra.run.smithi134.stderr:+ target_path=/dev/vg_nvme/lv_4 2022-03-20T15:00:41.207 INFO:teuthology.orchestra.run.smithi134.stderr:++ echo smithi134-lv-4 2022-03-20T15:00:41.208 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f1 -d - 2022-03-20T15:00:41.209 INFO:teuthology.orchestra.run.smithi134.stderr:+ host=smithi134 2022-03-20T15:00:41.209 INFO:teuthology.orchestra.run.smithi134.stderr:+ toolbox ceph orch device zap smithi134 /dev/vg_nvme/lv_4 --force 2022-03-20T15:00:41.209 INFO:teuthology.orchestra.run.smithi134.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi134 /dev/vg_nvme/lv_4 --force 2022-03-20T15:00:41.268 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:41.278 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:41.277916 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:00:41.473 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:41.472180 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-20T15:00:41.477 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:41.478779 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-20T15:00:41.726 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/vg_nvme/lv_4 on smithi134 zapped 2022-03-20T15:00:41.747 INFO:teuthology.orchestra.run.smithi134.stderr:+ zap_completion=0 2022-03-20T15:00:41.747 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' 0 = 0 ']' 2022-03-20T15:00:41.747 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-20T15:00:41.821 INFO:teuthology.orchestra.run.smithi134.stderr:+ zap_completion= 2022-03-20T15:00:41.822 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 3s 2022-03-20T15:00:42.176 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 25 16 7218 7202 18.0028 0 - 0.0339092 2022-03-20T15:00:43.177 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 26 16 7218 7202 17.3104 0 - 0.0339092 2022-03-20T15:00:43.488 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:43.488857 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-20T15:00:44.177 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 27 16 7218 7202 16.6693 0 - 0.0339092 2022-03-20T15:00:44.823 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' '' = 0 ']' 2022-03-20T15:00:44.823 INFO:teuthology.orchestra.run.smithi134.stderr:+ kubectl patch pv smithi134-lv-4 -p '{"spec":{"claimRef": null}}' 2022-03-20T15:00:45.177 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 28 16 7218 7202 16.074 0 - 0.0339092 2022-03-20T15:00:45.244 INFO:teuthology.orchestra.run.smithi134.stdout:persistentvolume/smithi134-lv-4 patched 2022-03-20T15:00:45.245 INFO:teuthology.orchestra.run.smithi134.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-20T15:00:45.245 INFO:teuthology.orchestra.run.smithi134.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-20T15:00:45.479 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:46.177 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 29 16 7582 7566 16.3041 1.75 0.0285302 0.0613102 2022-03-20T15:00:46.294 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:00:46.293553 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-20T15:00:46.295 INFO:tasks.rook.operator.smithi134.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-20T15:00:46.295 INFO:tasks.rook.operator.smithi134.stdout: Storage: v1.StorageScopeSpec{ 2022-03-20T15:00:46.295 INFO:tasks.rook.operator.smithi134.stdout: ... // 3 identical fields 2022-03-20T15:00:46.295 INFO:tasks.rook.operator.smithi134.stdout: Config: nil, 2022-03-20T15:00:46.296 INFO:tasks.rook.operator.smithi134.stdout: Selection: {}, 2022-03-20T15:00:46.296 INFO:tasks.rook.operator.smithi134.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-20T15:00:46.296 INFO:tasks.rook.operator.smithi134.stdout: {Name: "smithi134-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi134-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:46.296 INFO:tasks.rook.operator.smithi134.stdout: {Name: "smithi134-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi134-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:46.297 INFO:tasks.rook.operator.smithi134.stdout: {Name: "smithi134-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi134-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:46.297 INFO:tasks.rook.operator.smithi134.stdout:+ { 2022-03-20T15:00:46.297 INFO:tasks.rook.operator.smithi134.stdout:+ Name: "smithi134-lv-4", 2022-03-20T15:00:46.297 INFO:tasks.rook.operator.smithi134.stdout:+ Count: 1, 2022-03-20T15:00:46.298 INFO:tasks.rook.operator.smithi134.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-20T15:00:46.298 INFO:tasks.rook.operator.smithi134.stdout:+ }, 2022-03-20T15:00:46.298 INFO:tasks.rook.operator.smithi134.stdout: }, 2022-03-20T15:00:46.298 INFO:tasks.rook.operator.smithi134.stdout: }, 2022-03-20T15:00:46.298 INFO:tasks.rook.operator.smithi134.stdout: Annotations: nil, 2022-03-20T15:00:46.299 INFO:tasks.rook.operator.smithi134.stdout: Labels: nil, 2022-03-20T15:00:46.299 INFO:tasks.rook.operator.smithi134.stdout: ... // 20 identical fields 2022-03-20T15:00:46.299 INFO:tasks.rook.operator.smithi134.stdout: } 2022-03-20T15:00:46.300 INFO:teuthology.orchestra.run.smithi134.stdout:['Success'] 2022-03-20T15:00:46.338 INFO:teuthology.orchestra.run.smithi134.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-20T15:00:46.443 INFO:teuthology.orchestra.run.smithi134.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-20T15:00:46.496 INFO:teuthology.orchestra.run.smithi134.stdout:waiting for osd to come back up 2022-03-20T15:00:46.496 INFO:teuthology.orchestra.run.smithi134.stderr:+ num_osd=0 2022-03-20T15:00:46.497 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' 0 '!=' 4 ']' 2022-03-20T15:00:46.497 INFO:teuthology.orchestra.run.smithi134.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:00:46.497 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f3 -d ' ' 2022-03-20T15:00:46.498 INFO:teuthology.orchestra.run.smithi134.stderr:++ toolbox ceph osd stat 2022-03-20T15:00:46.498 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:00:46.654 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:47.177 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 30 16 8296 8280 17.248 44.625 0.00931543 0.0578557 2022-03-20T15:00:47.504 INFO:teuthology.orchestra.run.smithi134.stderr:+ num_osd=3 2022-03-20T15:00:47.504 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 30s 2022-03-20T15:00:48.177 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 31 16 9072 9056 18.2559 48.5 0.0197689 0.0547522 2022-03-20T15:00:49.177 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 32 16 9880 9864 19.2634 50.5 0.0190393 0.0517271 2022-03-20T15:00:50.178 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 33 16 10495 10479 19.8443 38.4375 0.0216669 0.0501689 2022-03-20T15:00:51.178 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 34 16 10902 10886 20.0087 25.4375 0.0126197 0.0499619 2022-03-20T15:00:52.178 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 35 16 11756 11740 20.9619 53.375 0.0279297 0.0476916 2022-03-20T15:00:53.178 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 36 16 12622 12606 21.8829 54.125 0.0140604 0.0456804 2022-03-20T15:00:54.178 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 37 16 13158 13142 22.1968 33.5 0.0227405 0.044657 2022-03-20T15:00:55.178 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 38 16 13813 13797 22.6899 40.9375 0.0226652 0.0440587 2022-03-20T15:00:56.179 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 39 16 14673 14657 23.4861 53.75 0.0149936 0.0425669 2022-03-20T15:00:57.179 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:00:57.179507+0000 min lat: 0.00126111 max lat: 13.4952 avg lat: 0.0413673 2022-03-20T15:00:57.179 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:00:57.179 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 40 16 15485 15469 24.1676 50.75 0.00907478 0.0413673 2022-03-20T15:00:58.179 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 41 16 16096 16080 24.5094 38.1875 0.0152819 0.0405365 2022-03-20T15:00:59.179 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 42 16 16794 16778 24.9644 43.625 0.0925409 0.0400432 2022-03-20T15:01:00.180 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 43 16 17548 17532 25.4793 47.125 0.0140057 0.0392349 2022-03-20T15:01:01.180 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 44 16 18179 18163 25.7965 39.4375 0.016438 0.0386575 2022-03-20T15:01:02.180 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 45 16 18565 18549 25.7593 24.125 0.00932675 0.03878 2022-03-20T15:01:03.180 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 46 16 19285 19269 26.1775 45 0.0164195 0.0381678 2022-03-20T15:01:04.180 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 47 16 19770 19754 26.2654 30.3125 0.0231695 0.0380584 2022-03-20T15:01:05.180 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 48 16 20329 20313 26.446 34.9375 0.0164877 0.0378025 2022-03-20T15:01:06.181 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 49 16 21058 21042 26.836 45.5625 0.0299496 0.0372531 2022-03-20T15:01:07.181 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 50 16 21777 21761 27.198 44.9375 0.0189436 0.0367577 2022-03-20T15:01:08.181 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 51 16 22566 22550 27.6315 49.3125 0.0143579 0.036182 2022-03-20T15:01:09.181 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 52 16 23179 23163 27.8368 38.3125 0.0884858 0.0358006 2022-03-20T15:01:10.181 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 53 16 23522 23506 27.716 21.4375 0.022832 0.0360701 2022-03-20T15:01:11.182 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 54 16 23919 23903 27.6622 24.8125 0.0206906 0.035885 2022-03-20T15:01:12.181 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 55 16 24184 24168 27.4604 16.5625 0.00370034 0.036038 2022-03-20T15:01:13.182 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 56 16 24184 24168 26.97 0 - 0.036038 2022-03-20T15:01:14.182 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 57 16 24184 24168 26.4969 0 - 0.036038 2022-03-20T15:01:15.182 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 58 16 24184 24168 26.04 0 - 0.036038 2022-03-20T15:01:16.182 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 59 16 24184 24168 25.5987 0 - 0.036038 2022-03-20T15:01:17.182 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:01:17.182070+0000 min lat: 0.00126111 max lat: 13.4952 avg lat: 0.036038 2022-03-20T15:01:17.183 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:01:17.183 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 60 16 24184 24168 25.172 0 - 0.036038 2022-03-20T15:01:17.507 INFO:teuthology.orchestra.run.smithi134.stdout:waiting for osd to come back up 2022-03-20T15:01:17.508 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' 3 '!=' 4 ']' 2022-03-20T15:01:17.508 INFO:teuthology.orchestra.run.smithi134.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:01:17.509 INFO:teuthology.orchestra.run.smithi134.stderr:++ toolbox ceph osd stat 2022-03-20T15:01:17.509 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:01:17.509 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f3 -d ' ' 2022-03-20T15:01:17.631 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:01:18.183 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 61 16 24184 24168 24.7594 0 - 0.036038 2022-03-20T15:01:18.478 INFO:teuthology.orchestra.run.smithi134.stderr:+ num_osd=3 2022-03-20T15:01:18.478 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 30s 2022-03-20T15:01:19.183 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 62 16 24184 24168 24.3601 0 - 0.036038 2022-03-20T15:01:20.183 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 63 16 24184 24168 23.9734 0 - 0.036038 2022-03-20T15:01:21.183 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 64 16 24184 24168 23.5988 0 - 0.036038 2022-03-20T15:01:22.183 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 65 16 24184 24168 23.2358 0 - 0.036038 2022-03-20T15:01:23.183 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 66 16 24184 24168 22.8837 0 - 0.036038 2022-03-20T15:01:24.184 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 67 16 24184 24168 22.5422 0 - 0.036038 2022-03-20T15:01:25.184 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 68 16 24184 24168 22.2107 0 - 0.036038 2022-03-20T15:01:26.184 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 69 16 24228 24212 21.9286 0.196429 0.0358292 0.0452688 2022-03-20T15:01:27.184 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 70 16 24228 24212 21.6154 0 - 0.0452688 2022-03-20T15:01:28.184 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 71 16 24228 24212 21.3109 0 - 0.0452688 2022-03-20T15:01:29.184 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 72 16 24228 24212 21.015 0 - 0.0452688 2022-03-20T15:01:30.184 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 73 16 24228 24212 20.7271 0 - 0.0452688 2022-03-20T15:01:31.185 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 74 16 24228 24212 20.447 0 - 0.0452688 2022-03-20T15:01:31.454 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:31.451631 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:01:31.454 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:31.451654 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:01:31.454 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:31.451663 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11187 quincy", pursuing 2022-03-20T15:01:31.919 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:31.916926 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:01:31.923 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:31.920075 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:01:31.924 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:31.920468 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:01:32.185 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 75 16 24228 24212 20.1744 0 - 0.0452688 2022-03-20T15:01:32.417 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:32.414855 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:01:32.669 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:32.666818 I | op-mon: start running mons 2022-03-20T15:01:33.020 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:33.018417 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:01:33.185 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 76 16 24228 24212 19.9089 0 - 0.0452688 2022-03-20T15:01:33.769 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:33.761541 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T15:01:33.780 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:33.776235 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:01:33.781 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:33.776405 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:01:33.799 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:33.795445 I | op-mon: targeting the mon count 1 2022-03-20T15:01:33.814 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:33.812034 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-03-20T15:01:33.814 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:33.812114 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:01:33.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:33.971170 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:01:34.185 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 77 16 24363 24347 19.76 1.05469 0.00637673 0.0503297 2022-03-20T15:01:34.841 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:34.828255 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:01:35.185 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 78 16 24363 24347 19.5066 0 - 0.0503297 2022-03-20T15:01:35.276 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:35.274166 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-20T15:01:35.308 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:35.306152 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-20T15:01:36.185 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 79 16 24363 24347 19.2597 0 - 0.0503297 2022-03-20T15:01:37.185 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:01:37.183924+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0503297 2022-03-20T15:01:37.186 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:01:37.186 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 80 16 24363 24347 19.019 0 - 0.0503297 2022-03-20T15:01:37.339 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:37.336088 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-20T15:01:38.186 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 81 16 24363 24347 18.7842 0 - 0.0503297 2022-03-20T15:01:39.186 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 82 16 24363 24347 18.5551 0 - 0.0503297 2022-03-20T15:01:39.345 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:39.343354 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-20T15:01:40.186 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 83 16 24363 24347 18.3315 0 - 0.0503297 2022-03-20T15:01:41.187 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 84 16 24363 24347 18.1133 0 - 0.0503297 2022-03-20T15:01:42.186 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 85 16 24730 24714 18.17 2.86719 0.012814 0.055027 2022-03-20T15:01:43.186 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 86 16 25404 25388 18.4485 42.125 0.0224652 0.0541946 2022-03-20T15:01:44.186 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 87 16 26121 26105 18.7515 44.8125 0.0128221 0.0533163 2022-03-20T15:01:45.187 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 88 16 26818 26802 19.0334 43.5625 0.0107533 0.0525307 2022-03-20T15:01:46.187 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 89 15 27647 27632 19.4023 51.875 0.0172634 0.0515303 2022-03-20T15:01:47.187 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 90 16 28517 28501 19.7902 54.3125 0.0159054 0.0505184 2022-03-20T15:01:48.187 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 91 16 29340 29324 20.1379 51.4375 0.0103757 0.0496235 2022-03-20T15:01:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' 3 '!=' 4 ']' 2022-03-20T15:01:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:01:48.481 INFO:teuthology.orchestra.run.smithi134.stdout:waiting for osd to come back up 2022-03-20T15:01:48.481 INFO:teuthology.orchestra.run.smithi134.stderr:++ toolbox ceph osd stat 2022-03-20T15:01:48.482 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:01:48.482 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f3 -d ' ' 2022-03-20T15:01:48.569 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:01:49.187 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 92 16 29877 29861 20.2838 33.5625 0.0219255 0.0492854 2022-03-20T15:01:50.087 INFO:teuthology.orchestra.run.smithi134.stderr:+ num_osd=3 2022-03-20T15:01:50.087 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 30s 2022-03-20T15:01:50.187 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 93 16 30505 30489 20.4876 39.25 0.0196461 0.0487675 2022-03-20T15:01:51.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 94 16 31199 31183 20.7311 43.375 0.0164518 0.0482278 2022-03-20T15:01:52.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 95 16 31974 31958 21.0227 48.4375 0.0224014 0.0475563 2022-03-20T15:01:53.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 96 16 32686 32670 21.2672 44.5 0.0269938 0.0470115 2022-03-20T15:01:54.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 97 16 33314 33298 21.4525 39.25 0.0383784 0.0466055 2022-03-20T15:01:55.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 98 16 34025 34009 21.687 44.4375 0.0131462 0.0461013 2022-03-20T15:01:55.623 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:55.620615 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:01:55.624 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:55.620655 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:01:55.624 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:55.620683 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11187 quincy", pursuing 2022-03-20T15:01:56.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 99 16 34559 34543 21.8051 33.375 0.0130281 0.0458504 2022-03-20T15:01:57.029 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:57.025483 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:01:57.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:01:57.185998+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0454696 2022-03-20T15:01:57.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:01:57.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 100 16 35202 35186 21.9888 40.1875 0.018653 0.0454696 2022-03-20T15:01:57.245 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:57.240927 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:01:57.245 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:57.241104 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:01:58.037 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:58.034053 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:01:58.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 101 16 35827 35811 22.1578 39.0625 0.0284641 0.045123 2022-03-20T15:01:59.013 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:59.009368 I | op-mon: start running mons 2022-03-20T15:01:59.082 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:01:59.078578 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:01:59.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 102 16 36483 36467 22.3425 41 0.0311295 0.0447423 2022-03-20T15:02:00.188 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 103 16 37074 37058 22.4842 36.9375 0.0136361 0.0444643 2022-03-20T15:02:00.343 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:00.339875 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T15:02:00.409 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:00.405842 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:02:00.410 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:00.406045 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:02:00.433 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:00.429009 I | op-mon: targeting the mon count 1 2022-03-20T15:02:00.436 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:00.433270 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:02:01.031 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:01.028160 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:02:01.032 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:01.028179 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:02:01.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 104 16 37745 37729 22.6712 41.9375 0.0211113 0.0440981 2022-03-20T15:02:01.574 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:01.571632 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:02:01.575 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:01.571651 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:02:01.916 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:01.913035 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:02:01.916 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:01.913057 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:02:02.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 105 16 37954 37938 22.5797 13.0625 0.00471166 0.043939 2022-03-20T15:02:02.251 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.248905 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:02:02.252 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.248922 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-20T15:02:02.630 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.626776 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-20T15:02:02.630 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.626800 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:02:02.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.970172 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:02:02.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.970204 I | op-mon: checking for basic quorum with existing mons 2022-03-20T15:02:02.974 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.970215 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-20T15:02:02.989 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.986420 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T15:02:02.995 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.992253 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:02:02.997 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:02.992419 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:02:03.006 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:03.003671 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:02:03.013 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:03.010798 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:02:03.014 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:03.010823 I | op-mon: waiting for mon quorum with [a] 2022-03-20T15:02:03.019 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:03.016153 I | op-mon: mons running: [a] 2022-03-20T15:02:03.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 106 16 37954 37938 22.3667 0 - 0.043939 2022-03-20T15:02:03.448 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:03.444728 I | op-mon: Monitors in quorum: [a] 2022-03-20T15:02:03.448 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:03.444760 I | op-mon: mons created: 1 2022-03-20T15:02:03.939 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:03.936150 I | op-mon: waiting for mon quorum with [a] 2022-03-20T15:02:03.948 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:03.944888 I | op-mon: mons running: [a] 2022-03-20T15:02:04.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 107 16 37954 37938 22.1576 0 - 0.043939 2022-03-20T15:02:04.351 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:04.348445 I | op-mon: Monitors in quorum: [a] 2022-03-20T15:02:04.352 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:04.348488 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T15:02:04.823 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:04.819559 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T15:02:05.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 108 16 37954 37938 21.9525 0 - 0.043939 2022-03-20T15:02:05.298 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:05.295216 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T15:02:05.816 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:05.813308 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T15:02:06.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 109 16 37954 37938 21.7511 0 - 0.043939 2022-03-20T15:02:06.247 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:06.243711 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T15:02:06.247 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:06.243732 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T15:02:06.719 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:06.715479 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T15:02:07.172 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:07.168579 I | cephclient: successfully enabled msgr2 protocol 2022-03-20T15:02:07.172 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:07.168604 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T15:02:07.172 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:07.168608 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T15:02:07.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 110 16 37954 37938 21.5533 0 - 0.043939 2022-03-20T15:02:07.595 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:07.592490 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T15:02:07.646 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:07.643611 I | op-mgr: start running mgr 2022-03-20T15:02:07.647 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:07.643655 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T15:02:08.155 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.152199 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T15:02:08.160 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.158059 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T15:02:08.163 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.160542 I | op-mgr: setting services to point to mgr "a" 2022-03-20T15:02:08.189 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 111 16 37954 37938 21.3591 0 - 0.043939 2022-03-20T15:02:08.299 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.296439 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T15:02:08.300 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.296466 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T15:02:08.320 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.316231 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T15:02:08.435 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.431481 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi134-lv-1" 2022-03-20T15:02:08.435 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.431516 I | op-osd: OSD PVC "smithi134-lv-1-data-02g9hz" already exists 2022-03-20T15:02:08.435 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.431526 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi134-lv-2" 2022-03-20T15:02:08.435 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.431534 I | op-osd: OSD PVC "smithi134-lv-2-data-0vt7mf" already exists 2022-03-20T15:02:08.436 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.431540 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi134-lv-3" 2022-03-20T15:02:08.436 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.431550 I | op-osd: OSD PVC "smithi134-lv-3-data-0nd6cl" already exists 2022-03-20T15:02:08.436 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.431559 I | op-osd: creating 1 new PVCs for device set "smithi134-lv-4" 2022-03-20T15:02:08.698 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.690318 I | op-osd: successfully provisioned PVC "smithi134-lv-4-data-0rvbkv" 2022-03-20T15:02:08.717 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.713206 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-1-data-02g9hz" 2022-03-20T15:02:08.718 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.713225 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-2-data-0vt7mf" 2022-03-20T15:02:08.718 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.713229 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-3-data-0nd6cl" 2022-03-20T15:02:08.719 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:08.713232 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-4-data-0rvbkv" 2022-03-20T15:02:09.050 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:09.046797 I | op-osd: started OSD provisioning job for PVC "smithi134-lv-4-data-0rvbkv" 2022-03-20T15:02:09.050 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:09.046815 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T15:02:09.050 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:09.046819 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T15:02:09.190 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 112 16 37954 37938 21.1684 0 - 0.043939 2022-03-20T15:02:09.276 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:09.272870 I | op-osd: OSD orchestration status for PVC smithi134-lv-4-data-0rvbkv is "starting" 2022-03-20T15:02:09.416 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:09.412678 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T15:02:09.428 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:09.420417 I | op-mgr: successful modules: prometheus 2022-03-20T15:02:09.431 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:09.425355 I | op-mgr: successful modules: dashboard 2022-03-20T15:02:09.775 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:09.771641 I | op-mgr: successful modules: balancer 2022-03-20T15:02:10.190 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 113 16 37954 37938 20.9811 0 - 0.043939 2022-03-20T15:02:10.808 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:10.804778 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-20T15:02:10.885 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:10.881102 I | op-osd: updating OSD 1 on PVC "smithi134-lv-2-data-0vt7mf" 2022-03-20T15:02:10.886 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:10.881129 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-2-data-0vt7mf" 2022-03-20T15:02:11.190 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 114 16 37954 37938 20.7971 0 - 0.043939 2022-03-20T15:02:12.027 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:12.024399 I | op-mgr: successful modules: orchestrator modules 2022-03-20T15:02:12.190 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 115 16 37954 37938 20.6162 0 - 0.043939 2022-03-20T15:02:12.361 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:12.358495 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-20T15:02:12.369 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:12.366016 I | op-osd: updating OSD 2 on PVC "smithi134-lv-3-data-0nd6cl" 2022-03-20T15:02:12.369 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:12.366046 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-3-data-0nd6cl" 2022-03-20T15:02:13.190 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 116 16 37954 37938 20.4385 0 - 0.043939 2022-03-20T15:02:13.595 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:13.592085 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-20T15:02:13.600 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:13.597126 I | op-osd: updating OSD 3 on PVC "smithi134-lv-1-data-02g9hz" 2022-03-20T15:02:13.600 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:13.597142 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-1-data-02g9hz" 2022-03-20T15:02:14.190 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 117 16 38266 38250 20.4305 1.625 0.0286384 0.0489388 2022-03-20T15:02:15.190 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 118 16 38942 38926 20.6153 42.25 0.0242808 0.048499 2022-03-20T15:02:16.190 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 119 16 39665 39649 20.8218 45.1875 0.0098798 0.048019 2022-03-20T15:02:17.190 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:02:17.188126+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0477068 2022-03-20T15:02:17.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:02:17.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 120 16 40241 40225 20.9482 36 0.0803035 0.0477068 2022-03-20T15:02:18.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 121 15 40735 40720 21.0308 30.9375 0.0156097 0.0475415 2022-03-20T15:02:19.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 122 16 41399 41383 21.1979 41.4375 0.020154 0.0471665 2022-03-20T15:02:20.094 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' 3 '!=' 4 ']' 2022-03-20T15:02:20.095 INFO:teuthology.orchestra.run.smithi134.stdout:waiting for osd to come back up 2022-03-20T15:02:20.096 INFO:teuthology.orchestra.run.smithi134.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:02:20.096 INFO:teuthology.orchestra.run.smithi134.stderr:++ toolbox ceph osd stat 2022-03-20T15:02:20.096 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:02:20.097 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f3 -d ' ' 2022-03-20T15:02:20.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 123 16 42081 42065 21.3721 42.625 0.0246416 0.0467835 2022-03-20T15:02:20.849 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:02:21.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 124 16 42727 42711 21.5253 40.375 0.0252975 0.0464472 2022-03-20T15:02:21.761 INFO:teuthology.orchestra.run.smithi134.stderr:+ num_osd=3 2022-03-20T15:02:21.761 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 30s 2022-03-20T15:02:22.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 125 16 43346 43330 21.6626 38.6875 0.0124144 0.0461551 2022-03-20T15:02:23.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 126 16 43997 43981 21.8135 40.6875 0.0194891 0.045834 2022-03-20T15:02:24.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 127 16 44662 44646 21.969 41.5625 0.0073646 0.0455094 2022-03-20T15:02:25.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 128 16 45328 45312 22.1225 41.625 0.0135497 0.0451934 2022-03-20T15:02:26.191 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 129 16 45931 45915 22.2432 37.6875 0.0248964 0.0449485 2022-03-20T15:02:27.192 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 130 16 46448 46432 22.3206 32.3125 0.0117129 0.0447893 2022-03-20T15:02:28.192 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 131 16 46932 46916 22.3811 30.25 0.0198043 0.0446721 2022-03-20T15:02:28.731 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:28.726861 I | op-osd: OSD orchestration status for node smithi134-lv-4-data-0rvbkv is "orchestrating" 2022-03-20T15:02:29.192 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 132 16 47562 47546 22.5098 39.375 0.0150507 0.044414 2022-03-20T15:02:29.342 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:29.333062 I | op-osd: OSD orchestration status for PVC smithi134-lv-4-data-0rvbkv is "orchestrating" 2022-03-20T15:02:30.192 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 133 16 48144 48128 22.614 36.375 0.0365664 0.0442108 2022-03-20T15:02:31.192 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 134 16 48738 48722 22.7223 37.125 0.0348399 0.0439999 2022-03-20T15:02:32.192 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 135 16 49324 49308 22.8253 36.625 0.0212427 0.0438028 2022-03-20T15:02:33.192 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 136 16 49860 49844 22.9037 33.5 0.0314284 0.0436532 2022-03-20T15:02:34.192 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 137 16 50402 50386 22.9838 33.875 0.0300737 0.0435012 2022-03-20T15:02:35.194 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 138 16 50982 50966 23.0796 36.25 0.0147361 0.0433169 2022-03-20T15:02:36.194 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 139 16 51578 51562 23.1815 37.25 0.0386641 0.0431288 2022-03-20T15:02:37.195 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:02:37.193287+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.043141 2022-03-20T15:02:37.196 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:02:37.196 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 140 16 51794 51778 23.1122 13.5 0.688046 0.043141 2022-03-20T15:02:37.266 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:37.263278 I | op-osd: OSD orchestration status for PVC smithi134-lv-4-data-0rvbkv is "completed" 2022-03-20T15:02:37.267 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:37.263308 I | op-osd: creating OSD 0 on PVC "smithi134-lv-4-data-0rvbkv" 2022-03-20T15:02:37.267 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:37.263319 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-4-data-0rvbkv" 2022-03-20T15:02:38.195 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 141 16 51852 51836 22.974 3.625 0.0171718 0.0435184 2022-03-20T15:02:39.196 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 142 16 52373 52357 23.0415 32.5625 0.0777282 0.0433867 2022-03-20T15:02:40.195 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 143 16 52807 52791 23.07 27.125 0.00314634 0.0432306 2022-03-20T15:02:41.017 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:40.991663 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T15:02:41.017 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:40.991686 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:02:41.196 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 144 16 52807 52791 22.9098 0 - 0.0432306 2022-03-20T15:02:41.224 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:41.213697 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:02:42.196 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 145 16 52807 52791 22.7518 0 - 0.0432306 2022-03-20T15:02:42.572 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:42.569669 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:02:43.051 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:43.048253 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:02:43.196 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 146 16 52807 52791 22.596 0 - 0.0432306 2022-03-20T15:02:44.196 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 147 16 52807 52791 22.4422 0 - 0.0432306 2022-03-20T15:02:45.196 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 148 16 52807 52791 22.2906 0 - 0.0432306 2022-03-20T15:02:46.196 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 149 16 52807 52791 22.141 0 - 0.0432306 2022-03-20T15:02:47.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 150 16 52807 52791 21.9934 0 - 0.0432306 2022-03-20T15:02:48.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 151 16 52807 52791 21.8477 0 - 0.0432306 2022-03-20T15:02:49.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 152 16 52807 52791 21.704 0 - 0.0432306 2022-03-20T15:02:50.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 153 16 52807 52791 21.5621 0 - 0.0432306 2022-03-20T15:02:51.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 154 16 52807 52791 21.4221 0 - 0.0432306 2022-03-20T15:02:51.762 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' 3 '!=' 4 ']' 2022-03-20T15:02:51.763 INFO:teuthology.orchestra.run.smithi134.stdout:waiting for osd to come back up 2022-03-20T15:02:51.763 INFO:teuthology.orchestra.run.smithi134.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:02:51.764 INFO:teuthology.orchestra.run.smithi134.stderr:++ toolbox ceph osd stat 2022-03-20T15:02:51.764 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f3 -d ' ' 2022-03-20T15:02:51.764 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:02:52.124 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:02:52.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 155 16 52807 52791 21.2839 0 - 0.0432306 2022-03-20T15:02:52.722 INFO:teuthology.orchestra.run.smithi134.stderr:+ num_osd=3 2022-03-20T15:02:52.723 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 30s 2022-03-20T15:02:53.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 156 16 52807 52791 21.1475 0 - 0.0432306 2022-03-20T15:02:54.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 157 16 52914 52898 21.0554 0.477679 0.0133754 0.0474773 2022-03-20T15:02:55.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 158 16 53034 53018 20.9696 7.5 0.00342363 0.0473919 2022-03-20T15:02:56.197 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 159 16 53588 53572 21.0554 34.625 0.0200735 0.0474849 2022-03-20T15:02:57.198 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:02:57.195718+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0473895 2022-03-20T15:02:57.198 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:02:57.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 160 16 54019 54003 21.0922 26.9375 0.0223766 0.0473895 2022-03-20T15:02:58.198 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 161 16 54560 54544 21.1712 33.8125 0.0312721 0.0472258 2022-03-20T15:02:58.381 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:58.377938 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:02:58.382 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:58.377957 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:02:58.382 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:58.377965 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11187 quincy", pursuing 2022-03-20T15:02:58.590 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:58.585103 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:02:58.674 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:58.672156 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:02:58.679 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:58.675688 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:02:59.198 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 162 16 55131 55115 21.2608 35.6875 0.0168322 0.0470252 2022-03-20T15:02:59.516 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:02:59.513495 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:03:00.198 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 163 16 55709 55693 21.3519 36.125 0.0222233 0.046825 2022-03-20T15:03:00.704 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:00.676233 I | op-mon: start running mons 2022-03-20T15:03:01.071 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:01.068374 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:03:01.198 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 164 16 56256 56240 21.4302 34.1875 0.0201269 0.0466525 2022-03-20T15:03:01.519 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:01.517135 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T15:03:02.198 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 165 16 56859 56843 21.5287 37.6875 0.0104601 0.0464421 2022-03-20T15:03:02.526 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:02.523018 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:03:02.528 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:02.523216 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:03:02.543 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:02.540782 I | op-mon: targeting the mon count 1 2022-03-20T15:03:02.551 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:02.547468 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:03:03.165 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:03.162500 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:03:03.166 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:03.162526 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:03:03.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 166 16 57261 57245 21.5503 25.125 0.0396009 0.0463942 2022-03-20T15:03:03.925 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:03.921225 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:03:03.926 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:03.921254 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:03:04.198 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 167 16 57779 57763 21.6151 32.375 0.0785547 0.0462397 2022-03-20T15:03:05.040 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:04.972653 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:03:05.051 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:04.972676 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:03:05.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 168 16 58305 58289 21.6821 32.875 0.0549449 0.0461094 2022-03-20T15:03:05.422 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:05.419376 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:03:05.422 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:05.419394 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-20T15:03:06.126 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:06.120004 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-03-20T15:03:06.127 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:06.120029 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:03:06.198 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 169 16 58760 58744 21.7221 28.4375 0.0281288 0.0460278 2022-03-20T15:03:06.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:06.687540 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:03:06.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:06.687565 I | op-mon: checking for basic quorum with existing mons 2022-03-20T15:03:06.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:06.687571 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-20T15:03:07.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 170 16 59347 59331 21.8101 36.6875 0.0170065 0.0458432 2022-03-20T15:03:07.311 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:07.306208 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.134:6789"]}] data:a=172.21.15.134:6789 mapping:{"node":{"a":{"Name":"smithi134","Hostname":"smithi134","Address":"172.21.15.134"}}} maxMonId:0] 2022-03-20T15:03:07.894 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:07.890417 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:03:07.894 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:07.890585 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:03:07.917 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:07.914138 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:03:07.933 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:07.930094 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:03:07.933 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:07.930171 I | op-mon: waiting for mon quorum with [a] 2022-03-20T15:03:07.939 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:07.936834 I | op-mon: mons running: [a] 2022-03-20T15:03:08.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 171 16 59769 59753 21.8368 26.375 0.0114713 0.0457834 2022-03-20T15:03:08.690 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:08.687584 I | op-mon: Monitors in quorum: [a] 2022-03-20T15:03:08.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:08.687708 I | op-mon: mons created: 1 2022-03-20T15:03:09.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 172 16 60287 60271 21.898 32.375 0.026065 0.0456564 2022-03-20T15:03:10.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 173 16 60825 60809 21.9658 33.625 0.0253011 0.0455176 2022-03-20T15:03:10.261 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:10.258551 I | op-mon: waiting for mon quorum with [a] 2022-03-20T15:03:10.959 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:10.956405 I | op-mon: mons running: [a] 2022-03-20T15:03:11.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 174 16 61304 61288 22.0116 29.9375 0.0253299 0.0454226 2022-03-20T15:03:12.079 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:12.076602 I | op-mon: Monitors in quorum: [a] 2022-03-20T15:03:12.080 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:12.076637 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T15:03:12.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 175 16 61897 61881 22.0976 37.0625 0.0129139 0.0452459 2022-03-20T15:03:12.821 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:12.818621 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T15:03:13.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 176 16 62410 62394 22.1542 32.0625 0.0190785 0.045129 2022-03-20T15:03:13.592 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:13.587541 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T15:03:14.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 177 16 62930 62914 22.2126 32.5 0.0184785 0.0450097 2022-03-20T15:03:14.283 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:14.280286 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T15:03:15.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 178 16 63429 63413 22.263 31.1875 0.100292 0.0449091 2022-03-20T15:03:15.299 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:15.296740 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T15:03:15.300 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:15.296760 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T15:03:16.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 179 16 63913 63897 22.3076 30.25 0.0206641 0.0448181 2022-03-20T15:03:16.238 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:16.235490 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T15:03:17.013 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:17.010225 I | cephclient: successfully enabled msgr2 protocol 2022-03-20T15:03:17.013 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:17.010251 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T15:03:17.014 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:17.010255 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T15:03:17.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:03:17.197677+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0446729 2022-03-20T15:03:17.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:03:17.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 180 16 64480 64464 22.3805 35.4375 0.0274528 0.0446729 2022-03-20T15:03:17.598 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:17.595493 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T15:03:17.625 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:17.622036 I | op-mgr: start running mgr 2022-03-20T15:03:17.625 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:17.622070 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T15:03:18.053 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.050199 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T15:03:18.061 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.059045 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T15:03:18.063 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.061341 I | op-mgr: setting services to point to mgr "a" 2022-03-20T15:03:18.191 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.185728 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T15:03:18.191 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.185758 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T15:03:18.199 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 181 16 64591 64575 22.2952 6.9375 0.0133782 0.0446488 2022-03-20T15:03:18.210 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.207874 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T15:03:18.215 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.210948 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi134-lv-1" 2022-03-20T15:03:18.215 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.211064 I | op-osd: OSD PVC "smithi134-lv-1-data-02g9hz" already exists 2022-03-20T15:03:18.215 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.211116 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi134-lv-2" 2022-03-20T15:03:18.216 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.211158 I | op-osd: OSD PVC "smithi134-lv-2-data-0vt7mf" already exists 2022-03-20T15:03:18.220 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.211223 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi134-lv-3" 2022-03-20T15:03:18.220 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.211264 I | op-osd: OSD PVC "smithi134-lv-3-data-0nd6cl" already exists 2022-03-20T15:03:18.220 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.211303 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi134-lv-4" 2022-03-20T15:03:18.220 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.211319 I | op-osd: OSD PVC "smithi134-lv-4-data-0rvbkv" already exists 2022-03-20T15:03:18.226 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.223542 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-1-data-02g9hz" 2022-03-20T15:03:18.226 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.223567 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-2-data-0vt7mf" 2022-03-20T15:03:18.227 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.223574 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-3-data-0nd6cl" 2022-03-20T15:03:18.227 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.223579 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-4-data-0rvbkv" 2022-03-20T15:03:18.227 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.223583 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T15:03:18.227 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:18.223587 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T15:03:19.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 182 16 64591 64575 22.1727 0 - 0.0446488 2022-03-20T15:03:19.478 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:19.475472 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-20T15:03:19.482 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:19.480248 I | op-osd: updating OSD 0 on PVC "smithi134-lv-4-data-0rvbkv" 2022-03-20T15:03:19.483 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:19.480274 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-4-data-0rvbkv" 2022-03-20T15:03:19.760 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:19.756828 I | op-mgr: successful modules: prometheus 2022-03-20T15:03:19.762 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:19.758707 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T15:03:19.766 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:19.764285 I | op-mgr: successful modules: dashboard 2022-03-20T15:03:19.972 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:19.968486 I | op-mgr: successful modules: balancer 2022-03-20T15:03:20.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 183 16 64591 64575 22.0515 0 - 0.0446488 2022-03-20T15:03:20.958 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:20.954782 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-20T15:03:20.968 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:20.964190 I | op-osd: updating OSD 1 on PVC "smithi134-lv-2-data-0vt7mf" 2022-03-20T15:03:20.969 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:20.964216 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-2-data-0vt7mf" 2022-03-20T15:03:21.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 184 16 64591 64575 21.9317 0 - 0.0446488 2022-03-20T15:03:21.446 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:21.443369 I | op-mgr: successful modules: orchestrator modules 2022-03-20T15:03:22.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 185 16 64594 64578 21.8142 0.046875 4.8088 0.0448699 2022-03-20T15:03:22.724 INFO:teuthology.orchestra.run.smithi134.stdout:waiting for osd to come back up 2022-03-20T15:03:22.724 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' 3 '!=' 4 ']' 2022-03-20T15:03:22.725 INFO:teuthology.orchestra.run.smithi134.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:03:22.725 INFO:teuthology.orchestra.run.smithi134.stderr:++ toolbox ceph osd stat 2022-03-20T15:03:22.725 INFO:teuthology.orchestra.run.smithi134.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:03:22.726 INFO:teuthology.orchestra.run.smithi134.stderr:++ cut -f3 -d ' ' 2022-03-20T15:03:22.832 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:22.821411 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-20T15:03:22.833 INFO:teuthology.orchestra.run.smithi134.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:03:22.835 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:22.830626 I | op-osd: updating OSD 2 on PVC "smithi134-lv-3-data-0nd6cl" 2022-03-20T15:03:22.836 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:22.830647 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-3-data-0nd6cl" 2022-03-20T15:03:23.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 186 16 64856 64840 21.7849 16.375 0.00798117 0.0458845 2022-03-20T15:03:23.683 INFO:teuthology.orchestra.run.smithi134.stderr:+ num_osd=4 2022-03-20T15:03:23.683 INFO:teuthology.orchestra.run.smithi134.stderr:+ sleep 30s 2022-03-20T15:03:24.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 187 16 65148 65132 21.766 18.25 0.0845583 0.0459288 2022-03-20T15:03:25.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 188 16 65453 65437 21.7516 19.0625 0.0454089 0.0459624 2022-03-20T15:03:25.367 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:25.358776 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-03-20T15:03:25.378 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:25.368770 I | op-osd: updating OSD 3 on PVC "smithi134-lv-1-data-02g9hz" 2022-03-20T15:03:25.379 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:25.368800 I | op-osd: OSD will have its main bluestore block on "smithi134-lv-1-data-02g9hz" 2022-03-20T15:03:26.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 189 16 65777 65761 21.7436 20.25 0.0398663 0.0459796 2022-03-20T15:03:26.243 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:26.239620 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T15:03:26.243 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:26.239644 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:03:27.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 190 16 65952 65936 21.6868 10.9375 0.0766487 0.0461003 2022-03-20T15:03:28.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 191 16 66154 66138 21.6393 12.625 0.0195037 0.0462019 2022-03-20T15:03:29.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 192 16 66525 66509 21.6474 23.1875 0.0243647 0.0461828 2022-03-20T15:03:30.200 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 193 16 66887 66871 21.6524 22.625 0.0403638 0.0461689 2022-03-20T15:03:31.201 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 194 16 67241 67225 21.6548 22.125 0.0272681 0.0461684 2022-03-20T15:03:32.201 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 195 16 67455 67439 21.6124 13.375 0.0369702 0.0461619 2022-03-20T15:03:33.201 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 196 16 67778 67762 21.6051 20.1875 0.111354 0.046273 2022-03-20T15:03:34.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 197 16 68132 68116 21.6075 22.125 0.0809898 0.0462681 2022-03-20T15:03:35.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 198 16 68484 68468 21.6095 22 0.0368125 0.0462642 2022-03-20T15:03:36.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 199 16 68835 68819 21.6112 21.9375 0.0978474 0.0462613 2022-03-20T15:03:37.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:03:37.201425+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0462804 2022-03-20T15:03:37.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:03:37.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 200 16 69147 69131 21.6006 19.5 0.0194301 0.0462804 2022-03-20T15:03:38.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 201 16 69458 69442 21.5898 19.4375 0.0210944 0.0462904 2022-03-20T15:03:39.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 202 16 69703 69687 21.5587 15.3125 0.0302623 0.0463731 2022-03-20T15:03:40.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 203 16 70063 70047 21.5634 22.5 0.0591407 0.0463661 2022-03-20T15:03:41.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 204 16 70405 70389 21.5624 21.375 0.03268 0.0463648 2022-03-20T15:03:42.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 205 16 70749 70733 21.5621 21.5 0.0382888 0.0463668 2022-03-20T15:03:43.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 206 16 70951 70935 21.5187 12.625 0.0403754 0.0464538 2022-03-20T15:03:44.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 207 16 71006 70990 21.4314 3.4375 0.00925559 0.0464787 2022-03-20T15:03:45.203 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 208 16 71006 70990 21.3283 0 - 0.0464787 2022-03-20T15:03:46.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 209 16 71219 71203 21.29 6.65625 0.0639857 0.0469588 2022-03-20T15:03:47.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 210 16 71496 71480 21.271 17.3125 0.107384 0.0469856 2022-03-20T15:03:48.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 211 16 71807 71791 21.2623 19.4375 0.077691 0.0470215 2022-03-20T15:03:49.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 212 16 72085 72069 21.244 17.375 0.0210233 0.0470553 2022-03-20T15:03:50.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 213 16 72420 72404 21.2425 20.9375 0.0462951 0.0470599 2022-03-20T15:03:51.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 214 16 72737 72721 21.2358 19.8125 0.0161157 0.0470796 2022-03-20T15:03:52.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 215 16 72916 72900 21.1891 11.1875 0.0893315 0.0470823 2022-03-20T15:03:53.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 216 16 72916 72900 21.091 0 - 0.0470823 2022-03-20T15:03:53.684 INFO:teuthology.orchestra.run.smithi134.stderr:+ '[' 4 '!=' 4 ']' 2022-03-20T15:03:53.686 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-20T15:03:53.698 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch status 2022-03-20T15:03:54.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 217 16 73137 73121 21.0575 6.90625 0.0521904 0.0474749 2022-03-20T15:03:54.825 INFO:teuthology.orchestra.run.smithi134.stdout:Backend: rook 2022-03-20T15:03:54.825 INFO:teuthology.orchestra.run.smithi134.stdout:Available: Yes 2022-03-20T15:03:54.875 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ps 2022-03-20T15:03:55.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 218 16 73435 73419 21.0463 18.625 0.0403704 0.0475025 2022-03-20T15:03:55.799 INFO:teuthology.orchestra.run.smithi134.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-20T15:03:55.800 INFO:teuthology.orchestra.run.smithi134.stdout:crashcollector.crash smithi134 running (6m) 0s ago 6m - - 7c0c85ada2e4 2022-03-20T15:03:55.800 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.a smithi134 running (8m) 0s ago 8m - - 7c0c85ada2e4 2022-03-20T15:03:55.800 INFO:teuthology.orchestra.run.smithi134.stdout:mon.a smithi134 running (8m) 0s ago 8m - - 7c0c85ada2e4 2022-03-20T15:03:55.800 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0 smithi134 running (75s) 0s ago 76s - - 7c0c85ada2e4 2022-03-20T15:03:55.800 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1 smithi134 running (6m) 0s ago 6m - - 7c0c85ada2e4 2022-03-20T15:03:55.801 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2 smithi134 running (6m) 0s ago 6m - - 7c0c85ada2e4 2022-03-20T15:03:55.801 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3 smithi134 running (6m) 0s ago 6m - - 7c0c85ada2e4 2022-03-20T15:03:55.845 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls 2022-03-20T15:03:56.204 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 219 16 73742 73726 21.0378 19.1875 0.0702588 0.0475221 2022-03-20T15:03:56.906 INFO:teuthology.orchestra.run.smithi134.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-20T15:03:56.907 INFO:teuthology.orchestra.run.smithi134.stdout:crash 1/1 0s ago 6m * 2022-03-20T15:03:56.907 INFO:teuthology.orchestra.run.smithi134.stdout:mgr 1/1 0s ago 8m count:1 2022-03-20T15:03:56.907 INFO:teuthology.orchestra.run.smithi134.stdout:mon 1/1 0s ago 8m count:1 2022-03-20T15:03:56.908 INFO:teuthology.orchestra.run.smithi134.stdout:osd 8 0s ago 6m 2022-03-20T15:03:56.908 INFO:teuthology.orchestra.run.smithi134.stdout:osd.all-available-devices 0 0s ago - * 2022-03-20T15:03:56.950 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch host ls 2022-03-20T15:03:57.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:03:57.203492+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0475411 2022-03-20T15:03:57.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:03:57.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 220 16 74047 74031 21.0288 19.0625 0.0751583 0.0475411 2022-03-20T15:03:57.758 INFO:teuthology.orchestra.run.smithi134.stdout:HOST ADDR LABELS STATUS 2022-03-20T15:03:57.758 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 172.21.15.134/smithi134 2022-03-20T15:03:57.758 INFO:teuthology.orchestra.run.smithi134.stdout:1 hosts in cluster 2022-03-20T15:03:57.793 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch device ls 2022-03-20T15:03:58.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 221 16 74340 74324 21.0165 18.3125 0.0357747 0.0475693 2022-03-20T15:03:58.868 INFO:teuthology.orchestra.run.smithi134.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-20T15:03:58.869 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:03:58.869 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:03:58.870 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:03:58.870 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:03:58.909 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch apply rgw foo 2022-03-20T15:03:59.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 222 15 74654 74639 21.0105 19.6875 0.0241245 0.0475825 2022-03-20T15:03:59.799 INFO:teuthology.orchestra.run.smithi134.stdout:Created 2022-03-20T15:03:59.803 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:03:59.801617 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-20T15:03:59.836 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch apply mds foo 2022-03-20T15:04:00.018 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:00.013514 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:04:00.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 223 16 74949 74933 20.9987 18.375 0.0269406 0.0476118 2022-03-20T15:04:00.851 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:00.849775 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-20T15:04:00.852 INFO:teuthology.orchestra.run.smithi134.stdout:Created 2022-03-20T15:04:00.871 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch apply rbd-mirror 2022-03-20T15:04:01.028 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:01.022936 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:04:01.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 224 16 75139 75123 20.9579 11.875 0.0203283 0.0476957 2022-03-20T15:04:01.857 INFO:teuthology.orchestra.run.smithi134.stdout:Success 2022-03-20T15:04:01.870 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:01.868415 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:04:01.896 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch apply nfs foo --port 12777 2022-03-20T15:04:02.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 225 16 75376 75360 20.9306 14.8125 0.0420249 0.0477612 2022-03-20T15:04:02.216 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:02.214559 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:04:02.515 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:02.489254 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-20T15:04:02.515 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:02.489276 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-20T15:04:02.516 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:02.489287 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-20T15:04:02.729 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:02.723317 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:04:02.730 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:02.723389 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:04:03.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 226 16 75498 75482 20.8717 7.625 0.0193766 0.0478995 2022-03-20T15:04:03.470 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:03.468036 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-20T15:04:03.637 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:03.626596 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:04:03.809 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:03.807940 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi134": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:04:04.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 227 16 75652 75636 20.8222 9.625 0.107257 0.0480116 2022-03-20T15:04:05.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 228 16 75891 75875 20.7964 14.9375 0.0319894 0.0480702 2022-03-20T15:04:06.205 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 229 16 76057 76041 20.7509 10.375 0.348224 0.0481759 2022-03-20T15:04:07.206 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 230 16 76235 76219 20.709 11.125 0.119982 0.0482742 2022-03-20T15:04:08.206 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 231 16 76465 76449 20.6816 14.375 0.043841 0.0483341 2022-03-20T15:04:08.480 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:08.478181 I | ceph-file-controller: creating filesystem "foo" 2022-03-20T15:04:09.206 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 232 16 76704 76688 20.6568 14.9375 0.0643957 0.0484014 2022-03-20T15:04:09.317 INFO:teuthology.orchestra.run.smithi134.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-20T15:04:09.317 INFO:teuthology.orchestra.run.smithi134.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-20T15:04:09.374 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2022-03-20T15:04:09.385 INFO:tasks.radosbench:joining radosbench (timing out after 11100s) 2022-03-20T15:04:09.385 INFO:teuthology.orchestra.run:waiting for 11100 2022-03-20T15:04:10.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 233 16 76965 76949 20.6382 16.3125 0.0393234 0.0484452 2022-03-20T15:04:11.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 234 16 77190 77174 20.6101 14.0625 0.0163566 0.0485083 2022-03-20T15:04:12.206 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 235 16 77377 77361 20.5721 11.6875 0.10339 0.0485991 2022-03-20T15:04:13.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 236 16 77606 77590 20.5455 14.3125 0.0259977 0.0486601 2022-03-20T15:04:14.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 237 16 77872 77856 20.529 16.625 0.102095 0.0486837 2022-03-20T15:04:15.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 238 16 78076 78060 20.4963 12.75 0.0918175 0.048774 2022-03-20T15:04:16.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 239 16 78360 78344 20.4848 17.75 0.01289 0.0487972 2022-03-20T15:04:16.730 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:16.721395 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.log" 2022-03-20T15:04:17.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:04:17.205945+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0488688 2022-03-20T15:04:17.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:04:17.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 240 16 78586 78570 20.4583 14.125 0.0122901 0.0488688 2022-03-20T15:04:18.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 241 16 78854 78838 20.4429 16.75 0.0320177 0.048906 2022-03-20T15:04:19.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 242 16 79163 79147 20.4382 19.3125 0.0818852 0.048914 2022-03-20T15:04:19.226 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:19.224450 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.non-ec" 2022-03-20T15:04:19.240 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:19.238180 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.control" 2022-03-20T15:04:19.284 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:19.281571 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.index" 2022-03-20T15:04:20.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 243 16 79411 79395 20.4179 15.5 0.065377 0.0489668 2022-03-20T15:04:21.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 244 16 79670 79654 20.4006 16.1875 0.0972596 0.0490006 2022-03-20T15:04:22.207 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 245 16 79910 79894 20.3785 15 0.145204 0.0490506 2022-03-20T15:04:22.768 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:22.766649 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.meta" 2022-03-20T15:04:22.771 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:22.767824 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-metadata" 2022-03-20T15:04:23.208 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 246 16 80102 80086 20.3444 12 0.00928171 0.049119 2022-03-20T15:04:24.211 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 247 16 80256 80240 20.301 9.625 0.0828162 0.0492351 2022-03-20T15:04:25.208 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 248 16 80540 80524 20.2907 17.75 0.0748423 0.0492731 2022-03-20T15:04:25.345 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:25.341973 I | cephclient: creating replicated pool foo.rgw.log succeeded 2022-03-20T15:04:25.345 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:25.343264 I | cephclient: creating replicated pool foo.rgw.control succeeded 2022-03-20T15:04:25.377 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:25.367656 I | cephclient: creating replicated pool foo-metadata succeeded 2022-03-20T15:04:25.399 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:25.397543 I | cephclient: creating replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-20T15:04:25.412 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:25.410146 I | cephclient: creating replicated pool foo.rgw.buckets.index succeeded 2022-03-20T15:04:26.208 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 249 16 80756 80740 20.2634 13.5 0.060314 0.0493373 2022-03-20T15:04:27.209 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 250 16 80994 80978 20.2419 14.875 0.0355348 0.0493704 2022-03-20T15:04:28.209 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 251 16 81272 81256 20.2304 17.375 0.0345238 0.0494211 2022-03-20T15:04:28.877 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:28.872902 I | cephclient: creating replicated pool foo.rgw.meta succeeded 2022-03-20T15:04:29.209 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 252 16 81464 81448 20.1978 12 0.167148 0.0494719 2022-03-20T15:04:29.858 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:29.854399 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:04:29.858 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:29.854503 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:04:29.911 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:29.909137 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:04:30.209 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 253 16 81706 81690 20.1777 15.125 0.0514347 0.0495471 2022-03-20T15:04:31.209 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 254 16 81988 81972 20.1676 17.625 0.0508369 0.0495753 2022-03-20T15:04:32.209 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 255 16 82276 82260 20.1591 18 0.0984885 0.0495827 2022-03-20T15:04:32.945 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:32.939755 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi134": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:04:33.040 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:33.016574 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-data0" 2022-03-20T15:04:33.209 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 256 16 82588 82572 20.1566 19.5 0.0230136 0.0495986 2022-03-20T15:04:33.270 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:33.206890 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:04:34.209 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 257 16 82905 82889 20.1552 19.8125 0.0314571 0.0496065 2022-03-20T15:04:35.209 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 258 16 83141 83125 20.1343 14.75 0.141259 0.0496498 2022-03-20T15:04:35.615 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:35.612752 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:04:35.616 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:35.612815 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:04:35.897 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:35.893078 I | cephclient: creating replicated pool foo-data0 succeeded 2022-03-20T15:04:35.897 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:35.893113 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-20T15:04:36.210 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 259 16 83497 83481 20.1424 22.25 0.0237102 0.0496365 2022-03-20T15:04:36.261 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:36.259860 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:04:37.210 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:04:37.208770+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0497104 2022-03-20T15:04:37.210 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:04:37.210 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 260 16 83691 83675 20.1116 12.125 0.0238929 0.0497104 2022-03-20T15:04:38.210 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 261 16 84007 83991 20.1102 19.75 0.00903288 0.0497135 2022-03-20T15:04:38.925 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:38.920214 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-20T15:04:39.210 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 262 16 84342 84326 20.1133 20.9375 0.0229982 0.049704 2022-03-20T15:04:39.633 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:39.631604 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:04:39.651 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:39.646438 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:04:39.697 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:39.695573 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-20T15:04:40.212 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 263 16 84680 84664 20.117 21.125 0.0506519 0.049698 2022-03-20T15:04:41.212 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 264 16 85005 84989 20.1177 20.3125 0.0188061 0.0496968 2022-03-20T15:04:41.229 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:41.227316 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-20T15:04:42.149 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:42.139537 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-20T15:04:42.205 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:42.202954 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:04:42.212 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 265 16 85347 85331 20.1225 21.375 0.0366473 0.0496862 2022-03-20T15:04:42.353 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:42.351140 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:04:42.353 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:42.351168 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:04:42.965 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:42.963564 I | op-mds: setting mds config flags 2022-03-20T15:04:42.966 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:42.963596 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:04:43.211 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:43.207393 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:04:43.212 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 266 16 85672 85656 20.1232 20.3125 0.158729 0.0496791 2022-03-20T15:04:43.822 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:43.819850 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:04:43.841 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:43.839312 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-20T15:04:44.212 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 267 16 85861 85845 20.0921 11.8125 0.042307 0.0497575 2022-03-20T15:04:45.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 268 16 86164 86148 20.0877 18.9375 0.0487047 0.0497707 2022-03-20T15:04:46.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 269 16 86287 86271 20.0416 7.6875 0.137166 0.0497799 2022-03-20T15:04:47.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 270 16 86320 86304 19.975 2.0625 0.0236781 0.0500182 2022-03-20T15:04:47.471 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:47.419083 I | op-mds: setting mds config flags 2022-03-20T15:04:47.472 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:47.419115 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:04:47.472 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:47.420463 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:04:47.472 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:47.445942 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:04:48.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 271 16 86654 86638 19.9784 20.875 0.0295222 0.0500438 2022-03-20T15:04:49.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 272 16 87019 87003 19.9888 22.8125 0.0414894 0.0500179 2022-03-20T15:04:49.248 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:49.246466 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:04:50.055 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:50.052861 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:04:50.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 273 16 87314 87298 19.9831 18.4375 0.0231134 0.0500293 2022-03-20T15:04:51.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 274 16 87674 87658 19.9923 22.5 0.0383092 0.0500082 2022-03-20T15:04:51.962 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:51.858873 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:04:51.962 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:51.858900 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:04:52.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 275 16 88009 87993 19.9957 20.9375 0.0127093 0.0500034 2022-03-20T15:04:53.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 276 16 88331 88315 19.9961 20.125 0.0460999 0.0499984 2022-03-20T15:04:53.344 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:53.333794 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:04:54.213 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 277 16 88553 88537 19.974 13.875 0.0751007 0.0500507 2022-03-20T15:04:55.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 278 16 88878 88862 19.9753 20.3125 0.0803829 0.0500512 2022-03-20T15:04:56.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 279 16 89199 89183 19.9756 20.0625 0.0760905 0.0500515 2022-03-20T15:04:57.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:57.173213 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:04:57.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:04:57.212914+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0500465 2022-03-20T15:04:57.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:04:57.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 280 16 89519 89503 19.9756 20 0.0269211 0.0500465 2022-03-20T15:04:58.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 281 16 89831 89815 19.9739 19.5 0.0692313 0.050057 2022-03-20T15:04:58.238 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:04:58.235414 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:04:59.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 282 16 90186 90170 19.9818 22.1875 0.101085 0.0500308 2022-03-20T15:05:00.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 283 16 90547 90531 19.9909 22.5625 0.0433565 0.0500137 2022-03-20T15:05:00.732 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:00.730128 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:05:01.062 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:01.052006 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:05:01.062 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:01.052032 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:05:01.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 284 16 90886 90870 19.9951 21.1875 0.0249058 0.0500024 2022-03-20T15:05:01.655 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:01.626977 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:05:02.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 285 16 91199 91183 19.9936 19.5625 0.0307984 0.0500057 2022-03-20T15:05:03.214 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 286 16 91471 91455 19.9831 17 0.106205 0.0500316 2022-03-20T15:05:04.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 287 16 91830 91814 19.9917 22.4375 0.0300363 0.0500098 2022-03-20T15:05:05.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 288 16 92192 92176 20.0008 22.625 0.0172374 0.0499876 2022-03-20T15:05:06.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 289 16 92570 92554 20.0133 23.625 0.026017 0.0499575 2022-03-20T15:05:07.109 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:07.107126 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:05:07.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 290 16 92878 92862 20.0107 19.25 0.0703472 0.0499622 2022-03-20T15:05:07.379 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:07.377586 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:05:08.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 291 16 93194 93178 20.0098 19.75 0.0266106 0.0499661 2022-03-20T15:05:09.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 292 16 93459 93443 19.998 16.5625 0.0416937 0.0499906 2022-03-20T15:05:09.689 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:09.684699 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:05:09.938 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:09.935148 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:05:09.941 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:09.935177 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:05:10.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 293 16 93794 93778 20.0012 20.9375 0.0621008 0.0499858 2022-03-20T15:05:10.785 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:10.780902 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:05:11.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 294 16 94085 94069 19.995 18.1875 0.0368145 0.0500007 2022-03-20T15:05:12.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 295 16 94378 94362 19.9893 18.3125 0.0446348 0.0500183 2022-03-20T15:05:13.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 296 16 94729 94713 19.9959 21.9375 0.0790264 0.0500008 2022-03-20T15:05:13.912 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:13.910012 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:05:14.079 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:14.077225 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:05:14.215 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 297 16 95102 95086 20.007 23.3125 0.0206229 0.04997 2022-03-20T15:05:15.216 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 298 16 95362 95346 19.9944 16.25 0.0452815 0.0500053 2022-03-20T15:05:16.216 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 299 16 95698 95682 19.9978 21 0.0324771 0.0499923 2022-03-20T15:05:16.650 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:16.648159 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:05:16.862 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:16.860756 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:05:16.867 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:16.861226 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:05:17.216 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:05:17.214901+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0499818 2022-03-20T15:05:17.216 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:05:17.216 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 300 16 96011 95995 19.9963 19.5625 0.0385212 0.0499818 2022-03-20T15:05:17.670 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:17.667887 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:05:18.216 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 301 16 96311 96295 19.9922 18.75 0.0386078 0.0500084 2022-03-20T15:05:19.216 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 302 16 96606 96590 19.987 18.4375 0.0456543 0.0500231 2022-03-20T15:05:20.217 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 303 16 96993 96977 20.0008 24.1875 0.0369118 0.0499878 2022-03-20T15:05:20.916 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:20.910826 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:05:21.217 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 304 16 97364 97348 20.0113 23.1875 0.0440638 0.0499614 2022-03-20T15:05:21.242 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:21.240549 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:05:22.217 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 305 16 97574 97558 19.9887 13.125 0.0241461 0.0499469 2022-03-20T15:05:23.105 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:23.103470 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:05:23.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:23.172582 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:05:23.175 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:23.172606 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:05:23.217 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 306 16 97882 97866 19.9863 19.25 0.0280508 0.0500225 2022-03-20T15:05:23.706 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:23.703675 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:05:24.218 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 307 16 98193 98177 19.9844 19.4375 0.097847 0.0500288 2022-03-20T15:05:25.218 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 308 16 98414 98398 19.9644 13.8125 0.0225887 0.0500778 2022-03-20T15:05:26.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 309 16 98801 98785 19.978 24.1875 0.0479051 0.0500454 2022-03-20T15:05:26.580 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:26.574997 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:05:27.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 310 16 99188 99172 19.9916 24.1875 0.0537954 0.0500093 2022-03-20T15:05:27.225 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:27.222900 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:05:28.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 311 16 99548 99532 19.9997 22.5 0.0795879 0.0499898 2022-03-20T15:05:29.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 312 16 99894 99878 20.0049 21.625 0.0156979 0.0499766 2022-03-20T15:05:29.336 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:29.335135 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:05:29.438 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:29.436477 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:05:29.438 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:29.436496 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:05:29.903 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:29.901649 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:05:30.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 313 16 100073 100057 19.9767 11.1875 0.0586284 0.0500445 2022-03-20T15:05:31.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 314 16 100216 100200 19.9415 8.9375 0.0107886 0.0501029 2022-03-20T15:05:32.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 315 16 100287 100271 19.8923 4.4375 0.0358594 0.0502615 2022-03-20T15:05:33.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 316 16 100546 100530 19.8806 16.1875 0.0694678 0.0502914 2022-03-20T15:05:33.902 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:33.900654 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:05:34.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 317 16 100894 100878 19.8865 21.75 0.0225991 0.0502752 2022-03-20T15:05:35.195 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:35.193742 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:05:35.219 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 318 16 101311 101295 19.9059 26.0625 0.0427269 0.0502277 2022-03-20T15:05:36.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 319 16 101671 101655 19.914 22.5 0.007483 0.050185 2022-03-20T15:05:36.884 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:36.882588 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:05:36.997 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:36.993462 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:05:36.997 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:36.993489 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:05:37.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:05:37.218804+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.050227 2022-03-20T15:05:37.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:05:37.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 320 16 101950 101934 19.9063 17.4375 0.0276262 0.050227 2022-03-20T15:05:37.623 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:37.621808 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:05:38.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 321 16 102280 102264 19.9085 20.625 0.0626701 0.050219 2022-03-20T15:05:39.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 322 16 102592 102576 19.9072 19.5 0.0171307 0.0502222 2022-03-20T15:05:40.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 323 16 102962 102946 19.9172 23.125 0.0738024 0.0501928 2022-03-20T15:05:41.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 324 16 103311 103295 19.923 21.8125 0.0149704 0.0501838 2022-03-20T15:05:41.542 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:41.540743 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:05:42.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 325 16 103675 103659 19.9317 22.75 0.0191917 0.0501588 2022-03-20T15:05:43.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 326 16 103965 103949 19.9262 18.125 0.024036 0.0501765 2022-03-20T15:05:44.220 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 327 16 104214 104198 19.9128 15.5625 0.00872747 0.050164 2022-03-20T15:05:44.738 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:44.646574 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:05:45.221 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 328 16 104214 104198 19.8521 0 - 0.050164 2022-03-20T15:05:46.221 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 329 16 104214 104198 19.7918 0 - 0.050164 2022-03-20T15:05:47.221 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 330 16 104418 104402 19.7704 4.25 0.0119321 0.0505707 2022-03-20T15:05:48.221 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 331 16 104807 104791 19.7841 24.3125 0.0700829 0.0505364 2022-03-20T15:05:48.332 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:48.329982 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:05:49.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 332 16 105164 105148 19.7917 22.3125 0.0161562 0.0505177 2022-03-20T15:05:50.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 333 16 105557 105541 19.806 24.5625 0.0718977 0.050475 2022-03-20T15:05:50.778 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:50.772757 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:05:50.779 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:50.772794 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:05:51.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 334 16 105906 105890 19.812 21.8125 0.0283366 0.0504666 2022-03-20T15:05:51.538 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:51.534277 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:05:52.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 335 16 106243 106227 19.8158 21.0625 0.0279388 0.0504526 2022-03-20T15:05:53.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 336 16 106565 106549 19.8167 20.125 0.0256749 0.0504525 2022-03-20T15:05:54.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 337 16 106868 106852 19.8141 18.9375 0.00809392 0.0504194 2022-03-20T15:05:54.398 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:54.396349 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:05:55.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 338 16 107239 107223 19.824 23.1875 0.0171317 0.0504359 2022-03-20T15:05:56.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 339 16 107541 107525 19.8212 18.875 0.0415913 0.0504411 2022-03-20T15:05:57.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:05:57.221474+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0503804 2022-03-20T15:05:57.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:05:57.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 340 16 107990 107974 19.8455 28.0625 0.0174229 0.0503804 2022-03-20T15:05:58.222 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 341 16 108432 108416 19.8683 27.625 0.0364827 0.050323 2022-03-20T15:05:59.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 342 16 108708 108692 19.8606 17.25 0.0197976 0.0503418 2022-03-20T15:05:59.538 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:05:59.535859 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:06:00.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 343 16 109089 109073 19.8721 23.8125 0.0566993 0.0503109 2022-03-20T15:06:01.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 344 16 109409 109393 19.8725 20 0.00999829 0.0502923 2022-03-20T15:06:01.813 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:01.811263 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:06:01.967 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:01.963713 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:06:01.972 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:01.963742 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:06:02.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 345 16 109631 109615 19.8551 13.875 0.0257962 0.0503567 2022-03-20T15:06:02.537 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:02.531562 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:06:03.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 346 16 109913 109897 19.8486 17.625 0.103798 0.0503683 2022-03-20T15:06:04.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 347 16 110230 110214 19.8485 19.8125 0.0448337 0.0503684 2022-03-20T15:06:05.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 348 16 110620 110604 19.8615 24.375 0.0097152 0.0503389 2022-03-20T15:06:05.634 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:05.632797 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:06:06.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 349 16 111067 111051 19.8847 27.9375 0.0114922 0.0502802 2022-03-20T15:06:07.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 350 16 111510 111494 19.9069 27.6875 0.0712111 0.0502236 2022-03-20T15:06:08.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 351 16 111968 111952 19.9318 28.625 0.0546491 0.0501612 2022-03-20T15:06:09.223 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 352 16 112342 112326 19.9415 23.375 0.0105152 0.050136 2022-03-20T15:06:10.224 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 353 16 112555 112539 19.9228 13.3125 0.00324009 0.050081 2022-03-20T15:06:11.224 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 354 16 112755 112739 19.9018 12.5 0.0320744 0.0502222 2022-03-20T15:06:12.224 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 355 16 112823 112807 19.8577 4.25 0.0695381 0.050347 2022-03-20T15:06:13.224 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 356 16 113153 113137 19.8599 20.625 0.0200873 0.0503434 2022-03-20T15:06:14.224 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 357 16 113421 113405 19.8511 16.75 0.0238164 0.050364 2022-03-20T15:06:15.224 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 358 16 113802 113786 19.8622 23.8125 0.046138 0.0503374 2022-03-20T15:06:15.882 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:15.880323 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:06:16.224 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 359 16 114208 114192 19.8775 25.375 0.0116314 0.0502982 2022-03-20T15:06:17.224 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:2022-03-20T15:06:17.223496+0000 min lat: 0.00126111 max lat: 14.8495 avg lat: 0.0502864 2022-03-20T15:06:17.225 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-03-20T15:06:17.225 INFO:tasks.radosbench.radosbench.a.smithi134.stdout: 360 5 114537 114532 19.8814 21.25 0.105763 0.0502864 2022-03-20T15:06:17.671 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:17.669779 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:06:18.225 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Total time run: 360.066 2022-03-20T15:06:18.225 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Total writes made: 114537 2022-03-20T15:06:18.225 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Write size: 65536 2022-03-20T15:06:18.226 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Object size: 65536 2022-03-20T15:06:18.226 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Bandwidth (MB/sec): 19.8813 2022-03-20T15:06:18.226 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Stddev Bandwidth: 16.2937 2022-03-20T15:06:18.226 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Max bandwidth (MB/sec): 95.125 2022-03-20T15:06:18.227 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Min bandwidth (MB/sec): 0 2022-03-20T15:06:18.227 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Average IOPS: 318 2022-03-20T15:06:18.227 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Stddev IOPS: 260.713 2022-03-20T15:06:18.227 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Max IOPS: 1522 2022-03-20T15:06:18.227 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Min IOPS: 0 2022-03-20T15:06:18.228 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Average Latency(s): 0.0502885 2022-03-20T15:06:18.228 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Stddev Latency(s): 0.373704 2022-03-20T15:06:18.228 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Max latency(s): 14.8495 2022-03-20T15:06:18.228 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Min latency(s): 0.00126111 2022-03-20T15:06:18.241 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Cleaning up (deleting benchmark objects) 2022-03-20T15:06:19.328 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:19.265580 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:06:19.329 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:19.265698 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:06:19.824 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:19.776497 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:06:23.686 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:23.684096 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:06:44.177 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:44.173772 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:06:46.151 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:46.149214 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:06:46.285 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:46.283788 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:06:46.285 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:46.283820 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:06:46.878 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:46.876455 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:06:49.860 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:06:49.854501 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:07:31.099 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:07:31.035921 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:07:33.525 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:07:33.523261 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:07:33.801 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:07:33.797167 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:07:33.801 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:07:33.797272 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:07:34.262 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:07:34.260192 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:07:36.886 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:07:36.884704 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:08:58.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:08:58.967898 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:09:01.298 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:09:01.288409 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:09:01.391 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:09:01.388908 I | ceph-object-controller: ceph object store gateway service running at 10.109.178.194 2022-03-20T15:09:01.396 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:09:01.388934 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:09:02.215 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:09:02.213523 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:09:05.018 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:09:05.016527 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-03-20T15:11:15.786 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Removed 114537 objects 2022-03-20T15:11:15.787 INFO:tasks.radosbench.radosbench.a.smithi134.stdout:Clean up completed and total clean up time :297.543 2022-03-20T15:11:16.466 INFO:tasks.rook.ceph_manager.ceph:removing pool_name unique_pool_0 2022-03-20T15:11:16.467 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2022-03-20T15:11:26.404 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_0' removed 2022-03-20T15:11:26.415 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-03-20T15:11:26.429 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-03-20T15:11:26.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-03-20T15:11:26.473 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-20T15:11:26.582 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:11:27.356 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:11:27.356 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:04:03.000000Z", "last_refresh": "2022-03-20T15:11:27.254716Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:27.254716Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:11:27.254716Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:11:27.254716Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:11:27.254716Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:27.254716Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-20T15:11:27.254716Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:27.254716Z", "running": 0, "size": 1}}] 2022-03-20T15:11:27.378 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch rm mds.foo 2022-03-20T15:11:27.913 INFO:teuthology.orchestra.run.smithi134.stdout:Removed cephfilesystems/foo 2022-03-20T15:11:27.934 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch rm rbd-mirror 2022-03-20T15:11:27.940 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:27.938632 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:11:28.477 INFO:teuthology.orchestra.run.smithi134.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-20T15:11:28.502 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch rm rgw.foo 2022-03-20T15:11:28.513 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:28.511736 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-20T15:11:29.136 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:29.133995 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-20T15:11:29.307 INFO:teuthology.orchestra.run.smithi134.stdout:Removed cephobjectstores/foo 2022-03-20T15:11:29.331 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:11:29.706 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:29.704205 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:11:31.238 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:31.232601 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-20T15:11:31.238 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:31.232632 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-20T15:11:31.238 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:31.232639 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-20T15:11:31.341 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:11:31.341 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:04:03.000000Z", "last_refresh": "2022-03-20T15:11:30.962840Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:30.962840Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:11:30.962840Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:11:30.962840Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:11:30.962840Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:30.962840Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:30.962840Z", "running": 0, "size": 1}}] 2022-03-20T15:11:33.245 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:33.237171 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-20T15:11:33.587 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:33.585063 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-20T15:11:34.493 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:34.466554 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-20T15:11:34.493 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:34.466582 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-20T15:11:34.494 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:34.466588 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-20T15:11:34.948 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:34.943506 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:11:34.928+0000 7fa045d95340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:11:34.949 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:11:34.928+0000 7fa045d95340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:11:34.949 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:11:34.928+0000 7fa045d95340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:11:34.949 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:11:34.950 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:34.943532 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:11:34.928+0000 7fa045d95340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:11:34.950 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:11:34.928+0000 7fa045d95340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:11:34.950 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:11:34.928+0000 7fa045d95340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:11:34.950 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:11:35.185 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:35.183648 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-20T15:11:35.186 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:35.183669 I | ceph-file-controller: downing filesystem "foo" 2022-03-20T15:11:37.551 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:37.549571 I | ceph-file-controller: downed filesystem "foo" 2022-03-20T15:11:41.430 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:11:42.604 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:11:42.605 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:04:03.000000Z", "last_refresh": "2022-03-20T15:11:41.917931Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:41.917931Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:11:41.917931Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:11:41.917931Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:11:41.917931Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:41.917931Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:41.917931Z", "running": 0, "size": 1}}] 2022-03-20T15:11:44.268 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:44.265926 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-20T15:11:44.271 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:44.266234 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-20T15:11:46.475 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:46.472886 I | cephclient: purge completed for pool "foo-metadata" 2022-03-20T15:11:46.882 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:46.880726 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-20T15:11:46.883 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:46.880744 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-20T15:11:48.757 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:48.753977 I | cephclient: purge completed for pool "foo-data0" 2022-03-20T15:11:48.757 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:48.754029 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-20T15:11:48.836 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:48.833888 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-20T15:11:48.869 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:48.867172 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-20T15:11:48.893 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:48.891374 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-20T15:11:48.899 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:48.891990 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:11:48.899 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:48.892442 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-20T15:11:49.477 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:49.474811 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:11:49.464+0000 7f925db01340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:11:49.478 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:11:49.464+0000 7f925db01340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:11:49.478 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:11:49.464+0000 7f925db01340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:11:49.478 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:11:49.478 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:49.474835 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:11:49.464+0000 7f925db01340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:11:49.479 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:11:49.464+0000 7f925db01340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:11:49.479 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:11:49.464+0000 7f925db01340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:11:49.479 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:11:52.613 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:11:52.611585 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi134": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:11:52.648 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:11:53.825 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:11:53.826 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:04:03.000000Z", "last_refresh": "2022-03-20T15:11:53.322100Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:11:53.322100Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:11:53.322100Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:11:53.322100Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:53.322100Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:53.322100Z", "running": 0, "size": 1}}] 2022-03-20T15:12:03.848 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:12:04.461 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:12:04.462 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:04:03.000000Z", "last_refresh": "2022-03-20T15:12:04.365232Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:12:04.365232Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:12:04.365232Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:12:04.365232Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:04.365232Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:04.365232Z", "running": 0, "size": 1}}] 2022-03-20T15:12:14.488 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:12:15.150 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:12:15.151 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:12:15.016490Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:12:15.016490Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:12:15.016490Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:12:15.016490Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:15.016490Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:15.016490Z", "running": 0, "size": 1}}] 2022-03-20T15:12:25.175 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:12:25.814 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:12:25.815 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:12:25.712612Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:12:25.712612Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:12:25.712612Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:12:25.712612Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:25.712612Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:25.712612Z", "running": 0, "size": 1}}] 2022-03-20T15:12:35.838 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:12:36.457 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:12:36.457 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:12:36.359102Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:12:36.359102Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:12:36.359102Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:12:36.359102Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:36.359102Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:36.359102Z", "running": 0, "size": 1}}] 2022-03-20T15:12:46.482 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:12:47.116 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:12:47.116 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:12:47.016224Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:12:47.016224Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:12:47.016224Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:12:47.016224Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:47.016224Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:47.016224Z", "running": 0, "size": 1}}] 2022-03-20T15:12:57.141 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:12:57.803 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:12:57.803 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:12:57.659281Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:12:57.659281Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:12:57.659281Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:12:57.659281Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:57.659281Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:57.659281Z", "running": 0, "size": 1}}] 2022-03-20T15:13:07.828 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:13:08.510 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:13:08.511 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:13:08.387490Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:13:08.387490Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:13:08.387490Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:13:08.387490Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:08.387490Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:08.387490Z", "running": 0, "size": 1}}] 2022-03-20T15:13:18.532 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:13:19.155 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:13:19.156 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:13:19.053551Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:13:19.053551Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:13:19.053551Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:13:19.053551Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:19.053551Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:19.053551Z", "running": 0, "size": 1}}] 2022-03-20T15:13:29.179 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:13:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:13:29.807 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:13:29.724906Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:13:29.724906Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:13:29.724906Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:13:29.724906Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:29.724906Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:29.724906Z", "running": 0, "size": 1}}] 2022-03-20T15:13:39.829 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:13:40.439 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:13:40.440 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:13:40.343873Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:13:40.343873Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:13:40.343873Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:13:40.343873Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:40.343873Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:40.343873Z", "running": 0, "size": 1}}] 2022-03-20T15:13:50.464 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:13:51.114 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:13:51.115 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:13:51.001593Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:13:51.001593Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:13:51.001593Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:13:51.001593Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:51.001593Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:51.001593Z", "running": 0, "size": 1}}] 2022-03-20T15:14:01.140 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:14:01.758 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:14:01.758 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:14:01.629026Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:14:01.629026Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:14:01.629026Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:14:01.629026Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:01.629026Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:01.629026Z", "running": 0, "size": 1}}] 2022-03-20T15:14:11.784 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:14:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:14:13.058 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:14:12.309136Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:14:12.309136Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:14:12.309136Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:14:12.309136Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:12.309136Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:12.309136Z", "running": 0, "size": 1}}] 2022-03-20T15:14:23.079 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:14:23.803 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:14:23.804 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:14:23.705962Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:14:23.705962Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:14:23.705962Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:14:23.705962Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:23.705962Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:23.705962Z", "running": 0, "size": 1}}] 2022-03-20T15:14:33.829 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:14:34.444 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:14:34.445 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:14:34.351458Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:14:34.351458Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:14:34.351458Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:14:34.351458Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:34.351458Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:34.351458Z", "running": 0, "size": 1}}] 2022-03-20T15:14:44.468 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:14:45.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:14:45.082 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:14:44.982114Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:14:44.982114Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:14:44.982114Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:14:44.982114Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:44.982114Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:44.982114Z", "running": 0, "size": 1}}] 2022-03-20T15:14:55.105 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:14:55.709 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:14:55.710 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:14:55.614152Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:14:55.614152Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:14:55.614152Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:14:55.614152Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:55.614152Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:55.614152Z", "running": 0, "size": 1}}] 2022-03-20T15:15:05.735 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:15:06.357 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:15:06.357 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:15:06.260816Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:15:06.260816Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:15:06.260816Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:15:06.260816Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:06.260816Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:06.260816Z", "running": 0, "size": 1}}] 2022-03-20T15:15:16.383 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:15:17.035 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:15:17.035 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:15:16.906682Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:15:16.906682Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:15:16.906682Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:15:16.906682Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:16.906682Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:16.906682Z", "running": 0, "size": 1}}] 2022-03-20T15:15:27.057 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:15:27.702 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:15:27.703 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:15:27.607888Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:15:27.607888Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:15:27.607888Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:15:27.607888Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:27.607888Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:27.607888Z", "running": 0, "size": 1}}] 2022-03-20T15:15:37.728 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:15:38.621 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:15:38.621 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:15:38.244714Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:15:38.244714Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:15:38.244714Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:15:38.244714Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:38.244714Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:38.244714Z", "running": 0, "size": 1}}] 2022-03-20T15:15:48.643 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:15:49.270 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:15:49.271 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:15:49.170230Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:15:49.170230Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:15:49.170230Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:15:49.170230Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:49.170230Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:49.170230Z", "running": 0, "size": 1}}] 2022-03-20T15:15:59.294 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:15:59.949 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:15:59.950 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:15:59.844296Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:15:59.844296Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:15:59.844296Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:15:59.844296Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:59.844296Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:59.844296Z", "running": 0, "size": 1}}] 2022-03-20T15:16:09.970 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:16:10.590 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:16:10.591 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:16:10.493286Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:16:10.493286Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:16:10.493286Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:16:10.493286Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:10.493286Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:10.493286Z", "running": 0, "size": 1}}] 2022-03-20T15:16:20.617 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:16:21.212 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:16:21.212 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:16:21.120901Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:16:21.120901Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:16:21.120901Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:16:21.120901Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:21.120901Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:21.120901Z", "running": 0, "size": 1}}] 2022-03-20T15:16:31.238 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:16:31.882 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:16:31.883 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:16:31.763319Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:16:31.763319Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:16:31.763319Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:16:31.763319Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:31.763319Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:31.763319Z", "running": 0, "size": 1}}] 2022-03-20T15:16:41.908 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:16:42.546 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:16:42.547 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:16:42.453664Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:16:42.453664Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:16:42.453664Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:16:42.453664Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:42.453664Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:42.453664Z", "running": 0, "size": 1}}] 2022-03-20T15:16:52.571 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:16:53.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:16:53.148 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:16:53.049227Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:16:53.049227Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:16:53.049227Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:16:53.049227Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:53.049227Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:53.049227Z", "running": 0, "size": 1}}] 2022-03-20T15:17:03.173 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:17:03.752 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:17:03.753 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:17:03.658237Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:17:03.658237Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:17:03.658237Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:17:03.658237Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:03.658237Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:03.658237Z", "running": 0, "size": 1}}] 2022-03-20T15:17:13.778 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:17:14.391 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:17:14.391 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:17:14.307606Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:17:14.307606Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:17:14.307606Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:17:14.307606Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:14.307606Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:14.307606Z", "running": 0, "size": 1}}] 2022-03-20T15:17:24.414 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:17:25.043 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:17:25.043 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:17:24.919693Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:17:24.919693Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:17:24.919693Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:17:24.919693Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:24.919693Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:24.919693Z", "running": 0, "size": 1}}] 2022-03-20T15:17:35.069 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:17:35.673 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:17:35.674 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:17:35.574153Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:17:35.574153Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:17:35.574153Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:17:35.574153Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:35.574153Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:35.574153Z", "running": 0, "size": 1}}] 2022-03-20T15:17:45.697 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:17:46.305 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:17:46.306 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:17:46.207257Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:17:46.207257Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:17:46.207257Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:17:46.207257Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:46.207257Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:46.207257Z", "running": 0, "size": 1}}] 2022-03-20T15:17:56.331 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:17:57.047 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:17:57.048 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:17:56.874784Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:17:56.874784Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:17:56.874784Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:17:56.874784Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:56.874784Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:56.874784Z", "running": 0, "size": 1}}] 2022-03-20T15:18:07.074 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:18:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:18:07.681 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:18:07.580634Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:18:07.580634Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:18:07.580634Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:18:07.580634Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:07.580634Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:07.580634Z", "running": 0, "size": 1}}] 2022-03-20T15:18:17.704 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:18:18.356 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:18:18.357 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:18:18.258743Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:18:18.258743Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:18:18.258743Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:18:18.258743Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:18.258743Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:18.258743Z", "running": 0, "size": 1}}] 2022-03-20T15:18:28.378 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:18:28.986 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:18:28.986 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:18:28.887940Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:18:28.887940Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:18:28.887940Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:18:28.887940Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:28.887940Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:28.887940Z", "running": 0, "size": 1}}] 2022-03-20T15:18:39.014 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:18:39.677 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:18:39.677 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:18:39.504453Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:18:39.504453Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:18:39.504453Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:18:39.504453Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:39.504453Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:39.504453Z", "running": 0, "size": 1}}] 2022-03-20T15:18:49.703 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:18:50.484 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:18:50.485 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:18:50.315604Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:18:50.315604Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:18:50.315604Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:18:50.315604Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:50.315604Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:50.315604Z", "running": 0, "size": 1}}] 2022-03-20T15:19:00.508 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:19:01.141 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:19:01.141 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:19:01.041702Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:19:01.041702Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:19:01.041702Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:19:01.041702Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:01.041702Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:01.041702Z", "running": 0, "size": 1}}] 2022-03-20T15:19:11.164 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:19:11.745 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:19:11.745 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:19:11.647815Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:19:11.647815Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:19:11.647815Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:19:11.647815Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:11.647815Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:11.647815Z", "running": 0, "size": 1}}] 2022-03-20T15:19:21.767 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:19:22.473 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:19:22.473 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:19:22.297790Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:19:22.297790Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:19:22.297790Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:19:22.297790Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:22.297790Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:22.297790Z", "running": 0, "size": 1}}] 2022-03-20T15:19:32.498 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:19:33.110 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:19:33.110 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:19:33.020660Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:19:33.020660Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:19:33.020660Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:19:33.020660Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:33.020660Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:33.020660Z", "running": 0, "size": 1}}] 2022-03-20T15:19:43.139 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:19:43.756 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:19:43.756 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:19:43.634290Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:19:43.634290Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:19:43.634290Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:19:43.634290Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:43.634290Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:43.634290Z", "running": 0, "size": 1}}] 2022-03-20T15:19:53.778 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:19:54.357 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:19:54.358 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:19:54.257653Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:19:54.257653Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:19:54.257653Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:19:54.257653Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:54.257653Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:54.257653Z", "running": 0, "size": 1}}] 2022-03-20T15:20:04.381 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:20:05.006 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:20:05.006 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:20:04.912134Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:20:04.912134Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:20:04.912134Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:20:04.912134Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:04.912134Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:04.912134Z", "running": 0, "size": 1}}] 2022-03-20T15:20:15.029 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:20:15.651 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:20:15.651 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:20:15.560577Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:20:15.560577Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:20:15.560577Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:20:15.560577Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:15.560577Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:15.560577Z", "running": 0, "size": 1}}] 2022-03-20T15:20:25.676 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:20:26.296 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:20:26.296 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:20:26.198483Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:20:26.198483Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:20:26.198483Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:20:26.198483Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:26.198483Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:26.198483Z", "running": 0, "size": 1}}] 2022-03-20T15:20:36.321 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:20:36.914 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:20:36.914 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:20:36.815517Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:20:36.815517Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:20:36.815517Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:20:36.815517Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:36.815517Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:36.815517Z", "running": 0, "size": 1}}] 2022-03-20T15:20:46.937 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:20:47.577 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:20:47.578 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:20:47.482940Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:20:47.482940Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:20:47.482940Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:20:47.482940Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:47.482940Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:47.482940Z", "running": 0, "size": 1}}] 2022-03-20T15:20:57.604 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:20:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:20:58.225 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:20:58.134834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:20:58.134834Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:20:58.134834Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:20:58.134834Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:58.134834Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:58.134834Z", "running": 0, "size": 1}}] 2022-03-20T15:21:08.247 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:21:08.891 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:21:08.891 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:21:08.784196Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:21:08.784196Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:21:08.784196Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:21:08.784196Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:08.784196Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:08.784196Z", "running": 0, "size": 1}}] 2022-03-20T15:21:18.915 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:21:19.649 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:21:19.649 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:21:19.562200Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:21:19.562200Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:21:19.562200Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:21:19.562200Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:19.562200Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:19.562200Z", "running": 0, "size": 1}}] 2022-03-20T15:21:29.674 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:21:33.341 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:21:33.341 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:21:33.234677Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:21:33.234677Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:21:33.234677Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:21:33.234677Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:33.234677Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:33.234677Z", "running": 0, "size": 1}}] 2022-03-20T15:21:43.365 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:21:44.009 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:21:44.009 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:21:43.913492Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:21:43.913492Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:21:43.913492Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:21:43.913492Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:43.913492Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:43.913492Z", "running": 0, "size": 1}}] 2022-03-20T15:21:54.030 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:21:54.663 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:21:54.663 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:21:54.565835Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:21:54.565835Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:21:54.565835Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:21:54.565835Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:54.565835Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:54.565835Z", "running": 0, "size": 1}}] 2022-03-20T15:22:04.685 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:22:05.303 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:22:05.303 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:22:05.208115Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:22:05.208115Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:22:05.208115Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:22:05.208115Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:05.208115Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:05.208115Z", "running": 0, "size": 1}}] 2022-03-20T15:22:15.329 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:22:15.993 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:22:15.993 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:22:15.863616Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:22:15.863616Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:22:15.863616Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:22:15.863616Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:15.863616Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:15.863616Z", "running": 0, "size": 1}}] 2022-03-20T15:22:26.015 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:22:26.637 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:22:26.637 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:22:26.536952Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:22:26.536952Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:22:26.536952Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:22:26.536952Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:26.536952Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:26.536952Z", "running": 0, "size": 1}}] 2022-03-20T15:22:36.660 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:22:37.260 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:22:37.261 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:22:37.165559Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:22:37.165559Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:22:37.165559Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:22:37.165559Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:37.165559Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:37.165559Z", "running": 0, "size": 1}}] 2022-03-20T15:22:44.846 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:22:44.844519 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:22:45.441 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:22:45.438821 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:22:45.429+0000 7fd0bfea1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:22:45.442 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:22:45.429+0000 7fd0bfea1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:22:45.442 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:22:45.429+0000 7fd0bfea1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:22:45.442 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:22:45.442 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:22:45.438846 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:22:45.429+0000 7fd0bfea1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:22:45.443 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:22:45.429+0000 7fd0bfea1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:22:45.443 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:22:45.429+0000 7fd0bfea1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:22:45.443 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:22:47.282 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:22:47.912 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:22:47.913 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:22:47.814340Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:22:47.814340Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:22:47.814340Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:22:47.814340Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:47.814340Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:47.814340Z", "running": 0, "size": 1}}] 2022-03-20T15:22:57.936 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:22:58.560 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:22:58.560 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:22:58.460456Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:22:58.460456Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:22:58.460456Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:22:58.460456Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:58.460456Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:58.460456Z", "running": 0, "size": 1}}] 2022-03-20T15:23:08.582 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:23:09.199 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:23:09.199 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:23:09.105880Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:23:09.105880Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:23:09.105880Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:23:09.105880Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:09.105880Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:09.105880Z", "running": 0, "size": 1}}] 2022-03-20T15:23:19.223 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:23:19.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:23:19.864 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:23:19.772290Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:23:19.772290Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:23:19.772290Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:23:19.772290Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:19.772290Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:19.772290Z", "running": 0, "size": 1}}] 2022-03-20T15:23:29.888 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:23:30.516 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:23:30.517 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:23:30.422227Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:23:30.422227Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:23:30.422227Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:23:30.422227Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:30.422227Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:30.422227Z", "running": 0, "size": 1}}] 2022-03-20T15:23:40.541 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:23:41.144 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:23:41.144 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:23:41.043281Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:23:41.043281Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:23:41.043281Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:23:41.043281Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:41.043281Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:41.043281Z", "running": 0, "size": 1}}] 2022-03-20T15:23:51.168 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:23:51.761 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:23:51.761 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:23:51.654829Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:23:51.654829Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:23:51.654829Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:23:51.654829Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:51.654829Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:51.654829Z", "running": 0, "size": 1}}] 2022-03-20T15:24:01.787 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:24:02.435 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:24:02.435 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:24:02.328329Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:24:02.328329Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:24:02.328329Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:24:02.328329Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:02.328329Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:02.328329Z", "running": 0, "size": 1}}] 2022-03-20T15:24:12.473 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:24:13.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:24:13.081 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:24:12.993934Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:24:12.993934Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:24:12.993934Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:24:12.993934Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:12.993934Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:12.993934Z", "running": 0, "size": 1}}] 2022-03-20T15:24:23.102 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:24:23.768 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:24:23.768 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:24:23.642362Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:24:23.642362Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:24:23.642362Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:24:23.642362Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:23.642362Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:23.642362Z", "running": 0, "size": 1}}] 2022-03-20T15:24:33.791 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:24:34.441 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:24:34.442 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:24:34.351736Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:24:34.351736Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:24:34.351736Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:24:34.351736Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:34.351736Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:34.351736Z", "running": 0, "size": 1}}] 2022-03-20T15:24:44.463 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:24:45.152 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:24:45.152 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:24:45.034707Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:24:45.034707Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:24:45.034707Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:24:45.034707Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:45.034707Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:45.034707Z", "running": 0, "size": 1}}] 2022-03-20T15:24:55.179 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:24:55.779 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:24:55.779 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:24:55.680841Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:24:55.680841Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:24:55.680841Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:24:55.680841Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:55.680841Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:55.680841Z", "running": 0, "size": 1}}] 2022-03-20T15:25:05.801 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:25:06.457 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:25:06.457 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:25:06.278882Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:25:06.278882Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:25:06.278882Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:25:06.278882Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:06.278882Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:06.278882Z", "running": 0, "size": 1}}] 2022-03-20T15:25:16.477 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:25:17.085 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:25:17.085 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:25:16.974734Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:25:16.974734Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:25:16.974734Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:25:16.974734Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:16.974734Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:16.974734Z", "running": 0, "size": 1}}] 2022-03-20T15:25:27.106 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:25:27.712 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:25:27.713 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:25:27.618375Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:25:27.618375Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:25:27.618375Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:25:27.618375Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:27.618375Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:27.618375Z", "running": 0, "size": 1}}] 2022-03-20T15:25:37.736 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:25:38.372 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:25:38.373 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:25:38.276423Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:25:38.276423Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:25:38.276423Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:25:38.276423Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:38.276423Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:38.276423Z", "running": 0, "size": 1}}] 2022-03-20T15:25:48.394 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:25:49.035 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:25:49.035 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:25:48.916838Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:25:48.916838Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:25:48.916838Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:25:48.916838Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:48.916838Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:48.916838Z", "running": 0, "size": 1}}] 2022-03-20T15:25:59.060 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:25:59.724 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:25:59.724 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:25:59.567256Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:25:59.567256Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:25:59.567256Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:25:59.567256Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:59.567256Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:59.567256Z", "running": 0, "size": 1}}] 2022-03-20T15:26:09.752 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:26:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:26:10.423 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:26:10.330324Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:26:10.330324Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:26:10.330324Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:26:10.330324Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:10.330324Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:10.330324Z", "running": 0, "size": 1}}] 2022-03-20T15:26:20.447 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:26:21.074 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:26:21.075 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:26:20.982446Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:26:20.982446Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:26:20.982446Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:26:20.982446Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:20.982446Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:20.982446Z", "running": 0, "size": 1}}] 2022-03-20T15:26:31.097 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:26:31.760 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:26:31.760 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:26:31.601870Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:26:31.601870Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:26:31.601870Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:26:31.601870Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:31.601870Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:31.601870Z", "running": 0, "size": 1}}] 2022-03-20T15:26:41.785 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:26:42.418 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:26:42.419 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:26:42.325187Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:26:42.325187Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:26:42.325187Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:26:42.325187Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:42.325187Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:42.325187Z", "running": 0, "size": 1}}] 2022-03-20T15:26:52.468 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:26:53.153 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:26:53.153 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:26:53.028913Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:26:53.028913Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:26:53.028913Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:26:53.028913Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:53.028913Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:53.028913Z", "running": 0, "size": 1}}] 2022-03-20T15:27:03.184 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:27:03.872 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:27:03.873 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:27:03.753631Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:27:03.753631Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:27:03.753631Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:27:03.753631Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:27:03.753631Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:27:03.753631Z", "running": 0, "size": 1}}] 2022-03-20T15:27:13.918 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:27:14.547 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:27:14.548 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:27:14.450854Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:27:14.450854Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:27:14.450854Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:27:14.450854Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:27:14.450854Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:27:14.450854Z", "running": 0, "size": 1}}] 2022-03-20T15:27:24.572 DEBUG:teuthology.orchestra.run.smithi134:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-xpmf2 -- ceph orch ls -f json 2022-03-20T15:27:25.196 INFO:teuthology.orchestra.run.smithi134.stdout: 2022-03-20T15:27:25.196 INFO:teuthology.orchestra.run.smithi134.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:11:52.000000Z", "last_refresh": "2022-03-20T15:27:25.101310Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:47.000000Z", "last_refresh": "2022-03-20T15:27:25.101310Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:55:16.000000Z", "last_refresh": "2022-03-20T15:27:25.101310Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:27:25.101310Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:27:25.101310Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:27:25.101310Z", "running": 0, "size": 1}}] 2022-03-20T15:27:25.215 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:27:25.216 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:27:25.217 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-20T15:27:25.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-20T15:27:25.231 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:27:25.231 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:27:25.255 DEBUG:teuthology.orchestra.remote:smithi134:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-03-20T15:27:25.269 DEBUG:teuthology.orchestra.run.smithi134:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-03-20T15:27:25.358 INFO:teuthology.orchestra.run.smithi134.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-20T15:27:25.371 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:27:25.372 DEBUG:teuthology.orchestra.run.smithi134:> kubectl delete -f cluster.yaml 2022-03-20T15:27:25.527 INFO:teuthology.orchestra.run.smithi134.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-20T15:27:25.529 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:25.528969 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-20T15:27:25.781 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:25.780448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:27:25.790 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:25.788987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:27:25.790 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:25.789021 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:27:36.010 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:36.008940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:27:36.020 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:36.019881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:27:46.279 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:46.278037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:27:46.290 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:46.289908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:27:56.519 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:56.518192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:27:56.533 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:27:56.532382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:06.771 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:06.769670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:06.783 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:06.780886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:16.999 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:16.997983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:17.014 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:17.013432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:27.272 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:27.271488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:27.285 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:27.284405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:37.538 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:37.536535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:37.546 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:37.545589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:47.764 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:47.763730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:47.774 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:47.773680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:57.991 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:57.989814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:28:58.004 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:28:58.003612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:08.226 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:08.224783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:08.238 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:08.237122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:18.473 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:18.471751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:18.484 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:18.483764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:28.713 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:28.711992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:29.129 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:29.127640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:39.348 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:39.346953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:39.363 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:39.362437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:49.601 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:49.600368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:49.615 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:49.614692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:59.833 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:59.832620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:29:59.846 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:29:59.845412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:10.067 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:10.065986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:10.077 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:10.076474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:20.305 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:20.303891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:20.316 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:20.315219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:30.595 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:30.594496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:30.611 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:30.610242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:40.831 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:40.830146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:40.842 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:40.840591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:51.062 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:51.060957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:30:51.085 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:30:51.084364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:01.307 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:01.306106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:01.318 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:01.317411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:11.547 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:11.544005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:11.557 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:11.556352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:21.777 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:21.776739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:21.789 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:21.788364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:32.007 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:32.005977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:32.019 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:32.018509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:42.239 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:42.238611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:42.251 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:42.251075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:52.474 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:52.473553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:31:52.487 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:31:52.486277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:02.713 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:02.712419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:02.726 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:02.725519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:12.945 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:12.944564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:12.955 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:12.954385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:23.175 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:23.174898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:23.190 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:23.189590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:33.413 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:33.412082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:33.429 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:33.428948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:43.655 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:43.653964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:43.668 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:43.667314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:53.895 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:53.886230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:32:53.901 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:32:53.899363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:04.140 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:04.139067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:04.157 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:04.157024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:14.431 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:14.430262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:14.443 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:14.441385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:24.668 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:24.667930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:24.678 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:24.678032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:34.904 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:34.903623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:34.915 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:34.915455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:45.136 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:45.134995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:45.146 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:45.146280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:55.365 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:55.364873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:33:55.417 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:33:55.417024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:05.636 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:05.636131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:05.650 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:05.650342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:15.871 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:15.870748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:15.884 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:15.883435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:26.102 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:26.101258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:26.113 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:26.113327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:36.336 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:36.335516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:36.349 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:36.348778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:46.571 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:46.570277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:46.582 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:46.581556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:56.805 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:56.804623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:34:56.819 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:34:56.819440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:07.042 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:07.041124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:07.051 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:07.051374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:17.272 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:17.271816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:17.287 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:17.286413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:27.506 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:27.505658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:27.560 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:27.559046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:37.786 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:37.785286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:37.798 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:37.798181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:48.021 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:48.020191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:48.034 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:48.033734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:58.251 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:58.250343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:35:58.264 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:35:58.263825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:08.487 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:08.485915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:08.498 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:08.498191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:18.718 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:18.717615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:18.731 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:18.730380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:28.952 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:28.951439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:28.965 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:28.964683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:39.185 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:39.184237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:39.196 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:39.196016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:49.419 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:49.418729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:49.432 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:49.431645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:59.652 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:59.651548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:36:59.664 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:36:59.663578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:09.922 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:09.922043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:09.932 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:09.931700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:20.154 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:20.151686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:20.168 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:20.164854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:30.407 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:30.406816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:30.463 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:30.462327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:40.685 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:40.684422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:40.699 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:40.698768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:50.959 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:50.958062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:37:50.968 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:37:50.966992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:01.214 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:01.213987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:01.226 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:01.223388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:11.462 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:11.461458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:11.475 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:11.474723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:21.692 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:21.691603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:21.705 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:21.704441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:31.929 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:31.927797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:31.944 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:31.943730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:42.169 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:42.168081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:42.181 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:42.180911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:52.402 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:52.401722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:52.413 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:52.412702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:38:52.668 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:38:52.667966 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 15:38:52.657328318 +0000 UTC m=+2681.378617114 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:39:02.629 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:02.628839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:02.639 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:02.638587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:12.858 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:12.857305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:12.872 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:12.871346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:23.100 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:23.099871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:23.114 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:23.111983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:25.447 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:25.447064 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:39:26.045 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:26.043704 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:39:26.031+0000 7f0815829340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:39:26.045 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:39:26.031+0000 7f0815829340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:39:26.046 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:39:26.031+0000 7f0815829340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:39:26.046 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:39:26.046 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:26.043730 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:39:26.031+0000 7f0815829340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:39:26.046 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:39:26.031+0000 7f0815829340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:39:26.047 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:39:26.031+0000 7f0815829340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:39:26.047 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:39:33.341 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:33.340381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:33.354 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:33.353969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:43.641 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:43.640943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:43.653 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:43.652469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:53.892 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:53.891080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:39:53.911 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:39:53.910938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:04.156 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:04.155929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:04.169 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:04.167488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:14.630 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:14.629750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:14.642 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:14.641561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:24.951 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:24.950723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:24.960 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:24.960105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:35.206 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:35.205478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:35.219 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:35.218788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:45.607 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:45.606524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:45.618 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:45.617640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:55.844 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:55.843576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:40:55.856 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:40:55.855793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:06.074 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:06.073495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:06.089 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:06.088907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:16.364 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:16.363485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:16.392 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:16.391414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:26.631 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:26.628481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:26.674 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:26.673602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:36.895 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:36.894612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:36.906 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:36.906323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:47.139 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:47.138031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:47.149 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:47.148570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:57.368 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:57.367379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:41:57.424 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:41:57.423364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:07.680 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:07.679165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:07.690 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:07.689336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:17.947 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:17.946387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:17.956 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:17.955755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:28.172 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:28.171415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:28.184 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:28.183084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:38.406 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:38.405489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:38.417 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:38.416564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:48.828 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:48.828022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:48.840 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:48.839182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:59.089 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:59.088540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:42:59.104 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:42:59.103417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:09.321 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:09.320916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:09.333 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:09.332734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:19.555 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:19.553963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:19.571 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:19.566775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:29.788 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:29.787463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:29.801 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:29.801005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:40.052 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:40.051545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:40.070 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:40.069531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:50.421 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:50.420836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:43:50.432 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:43:50.432354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:00.654 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:00.652425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:00.663 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:00.662854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:10.883 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:10.882437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:10.897 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:10.897444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:21.122 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:21.121956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:21.137 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:21.136199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:31.357 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:31.356077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:31.369 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:31.368167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:41.588 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:41.587886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:41.631 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:41.630528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:51.914 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:51.913428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:44:51.927 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:44:51.926831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:02.148 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:02.147549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:02.161 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:02.160464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:12.427 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:12.421949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:12.553 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:12.436196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:22.708 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:22.704116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:22.712 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:22.712260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:32.929 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:32.928358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:32.938 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:32.937804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:43.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:43.173259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:43.186 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:43.185580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:53.405 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:53.405019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:45:53.418 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:45:53.416718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:03.658 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:03.657396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:03.670 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:03.669781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:13.891 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:13.887651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:13.898 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:13.898002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:24.120 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:24.119366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:24.131 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:24.130771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:34.350 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:34.349219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:34.361 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:34.360433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:44.585 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:44.584373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:44.596 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:44.595654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:54.816 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:54.815639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:46:54.825 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:46:54.825167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:05.048 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:05.047446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:05.062 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:05.061093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:15.279 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:15.278753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:15.294 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:15.293962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:25.524 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:25.523297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:25.537 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:25.536206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:35.755 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:35.754188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:35.768 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:35.767618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:45.985 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:45.984834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:45.996 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:45.996375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:56.218 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:56.217110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:47:56.230 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:47:56.229971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:06.449 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:06.448842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:06.464 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:06.463795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:16.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:16.690348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:16.700 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:16.699966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:26.964 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:26.963514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:26.979 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:26.979318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:37.254 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:37.253677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:37.266 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:37.265832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:47.484 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:47.483450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:47.496 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:47.495279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:57.720 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:57.719358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:48:57.762 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:48:57.761582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:07.981 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:07.980314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:07.994 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:07.993714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:18.215 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:18.214745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:18.229 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:18.228719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:28.480 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:28.479277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:28.493 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:28.492528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:38.715 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:38.714556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:38.724 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:38.723412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:48.942 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:48.941458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:48.954 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:48.953984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:59.175 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:59.174043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:49:59.192 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:49:59.191935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:09.411 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:09.410501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:09.422 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:09.421561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:19.650 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:19.649638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:19.664 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:19.663241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:29.885 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:29.884051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:29.940 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:29.939410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:40.184 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:40.183138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:40.194 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:40.193758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:50.422 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:50.421064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:50:50.433 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:50:50.432721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:00.652 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:00.651064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:00.663 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:00.662880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:10.883 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:10.882120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:10.892 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:10.891462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:21.113 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:21.112012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:21.125 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:21.124281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:31.344 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:31.343017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:31.357 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:31.356700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:41.579 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:41.578124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:41.590 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:41.589483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:51.812 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:51.811157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:51:51.823 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:51:51.823070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:02.046 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:02.045111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:02.057 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:02.057113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:12.280 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:12.279092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:12.293 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:12.293317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:22.521 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:22.520148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:22.532 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:22.531250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:32.775 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:32.774603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:32.797 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:32.796810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:43.014 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:43.013782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:43.025 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:43.025158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:53.241 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:53.239887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:52:53.249 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:52:53.248437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:03.470 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:03.469490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:03.483 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:03.483129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:13.704 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:13.703560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:13.716 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:13.715455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:23.937 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:23.936299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:23.951 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:23.949845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:34.167 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:34.166364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:34.210 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:34.210189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:44.428 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:44.426832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:44.439 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:44.438391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:54.659 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:54.657876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:53:54.670 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:53:54.669509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:04.891 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:04.889814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:04.902 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:04.901446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:15.139 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:15.137911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:15.152 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:15.151404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:25.373 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:25.372269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:25.385 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:25.384685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:35.612 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:35.610926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:35.624 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:35.623367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:45.843 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:45.842057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:45.854 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:45.853463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:56.076 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:56.075172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:54:56.104 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:54:56.102859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:06.324 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:06.323756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:06.339 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:06.338259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:06.711 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:06.710176 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 15:55:06.703069453 +0000 UTC m=+3655.424358226 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:55:16.579 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:16.578578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:16.592 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:16.591683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:26.828 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:26.828120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:26.840 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:26.839957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:37.105 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:37.104074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:37.118 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:37.117183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:47.336 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:47.335514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:47.347 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:47.347254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:57.567 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:57.566676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:55:57.579 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:55:57.578608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:06.053 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:06.052549 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T15:56:06.642 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:06.640576 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:56:06.631+0000 7f461598d340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:56:06.642 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:56:06.631+0000 7f461598d340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:56:06.642 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:56:06.631+0000 7f461598d340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:56:06.643 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:56:06.643 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:06.640600 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:56:06.631+0000 7f461598d340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:56:06.643 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:56:06.631+0000 7f461598d340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:56:06.643 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T15:56:06.631+0000 7f461598d340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:56:06.644 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:56:07.627 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:07.626332 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 15:56:07.616402504 +0000 UTC m=+3716.337691251 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:56:07.799 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:07.798531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:07.811 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:07.811273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:18.030 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:18.029759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:18.043 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:18.042386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:28.265 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:28.263811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:28.277 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:28.276689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:38.497 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:38.495853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:38.508 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:38.507590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:48.776 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:48.775292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:48.785 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:48.784534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:59.009 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:59.008330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:56:59.048 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:56:59.044638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:09.265 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:09.264197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:09.277 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:09.276745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:19.505 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:19.504255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:19.518 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:19.517091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:29.739 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:29.738508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:29.753 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:29.752944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:39.969 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:39.968789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:39.983 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:39.982030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:50.231 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:50.229878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:57:50.241 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:57:50.240665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:00.460 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:00.459525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:00.472 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:00.471322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:10.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:10.690068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:10.700 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:10.699316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:20.917 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:20.916041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:20.932 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:20.931665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:31.152 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:31.151681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:31.164 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:31.164056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:41.383 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:41.382551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:41.395 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:41.394639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:51.614 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:51.613621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:58:51.629 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:58:51.628305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:01.849 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:01.848154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:01.861 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:01.861262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:12.095 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:12.093913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:12.107 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:12.107114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:22.360 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:22.359450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:22.383 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:22.382768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:32.603 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:32.601612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:32.614 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:32.613569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:42.839 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:42.832813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:42.854 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:42.852382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:53.096 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:53.095287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T15:59:53.106 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 15:59:53.105820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:03.329 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:03.328227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:03.342 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:03.341151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:13.564 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:13.562914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:13.578 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:13.577825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:23.800 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:23.799015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:23.813 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:23.811960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:34.065 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:34.064120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:34.077 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:34.076379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:44.297 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:44.296193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:44.306 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:44.305726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:54.525 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:54.524007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:00:54.604 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:00:54.603509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:04.824 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:04.823065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:04.835 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:04.834705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:15.054 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:15.053146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:15.068 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:15.067467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:25.291 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:25.289884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:25.302 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:25.301711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:35.522 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:35.521455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:35.534 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:35.533499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:45.778 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:45.777695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:45.792 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:45.791218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:56.031 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:56.030019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:01:56.041 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:01:56.040897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:06.267 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:06.266062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:06.276 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:06.275604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:16.497 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:16.496522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:16.509 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:16.508615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:26.730 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:26.729489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:26.739 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:26.738649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:36.962 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:36.961099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:36.974 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:36.973933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:47.195 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:47.194340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:47.206 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:47.205611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:57.426 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:57.425506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:02:57.440 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:02:57.439573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:07.661 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:07.659672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:07.674 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:07.673921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:17.893 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:17.892394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:17.906 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:17.905854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:28.126 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:28.125318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:28.138 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:28.137156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:38.359 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:38.358226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:38.370 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:38.369986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:48.602 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:48.601903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:48.613 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:48.613169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:58.833 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:58.832033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:03:58.842 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:03:58.842386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:09.062 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:09.061187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:09.073 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:09.073062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:19.463 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:19.462663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:19.473 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:19.472360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:29.696 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:29.695488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:29.707 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:29.707178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:39.927 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:39.926104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:39.986 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:39.985631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:50.452 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:50.450984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:04:50.463 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:04:50.462806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:00.681 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:00.680790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:00.695 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:00.692924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:10.915 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:10.914128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:10.923 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:10.923308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:21.141 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:21.140095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:21.151 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:21.150613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:31.373 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:31.372216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:31.387 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:31.386478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:41.614 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:41.613509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:41.627 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:41.625343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:51.845 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:51.844229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:05:51.857 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:05:51.856818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:02.078 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:02.077535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:02.091 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:02.090902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:12.311 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:12.310537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:12.323 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:12.322264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:22.580 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:22.579244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:22.592 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:22.591507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:32.813 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:32.811943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:32.825 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:32.825138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:43.039 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:43.038789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:43.050 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:43.049821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:53.275 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:53.274675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:06:53.294 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:06:53.293433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:03.524 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:03.523124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:03.537 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:03.536210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:13.768 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:13.766873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:13.782 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:13.781483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:24.058 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:24.057834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:24.068 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:24.067159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:34.285 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:34.283777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:34.299 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:34.297505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:44.520 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:44.519551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:44.533 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:44.532238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:54.754 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:54.753403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:07:54.765 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:07:54.764565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:04.989 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:04.988178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:05.000 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:05.000130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:15.215 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:15.214384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:15.264 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:15.262264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:25.490 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:25.489013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:25.501 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:25.500827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:35.740 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:35.739538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:35.754 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:35.753498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:45.994 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:45.993947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:46.005 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:46.004360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:56.235 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:56.234199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:08:56.244 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:08:56.244220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:06.465 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:06.464230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:06.475 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:06.474940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:16.697 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:16.695605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:16.710 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:16.709216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:26.963 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:26.962436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:26.976 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:26.975412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:37.194 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:37.193545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:37.203 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:37.202957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:47.428 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:47.427140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:47.438 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:47.437407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:57.669 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:57.668595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:09:57.680 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:09:57.679681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:07.899 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:07.897677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:07.910 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:07.909402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:18.164 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:18.163431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:18.176 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:18.175751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:28.395 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:28.394148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:28.406 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:28.406091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:38.627 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:38.626456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:38.639 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:38.638660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:48.884 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:48.882943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:48.898 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:48.896748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:59.114 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:59.112567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:10:59.126 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:10:59.125268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:09.347 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:09.345908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:09.356 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:09.355841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:19.574 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:19.573420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:19.586 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:19.585744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:29.804 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:29.803674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:29.818 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:29.817349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:40.036 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:40.035941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:40.048 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:40.047149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:50.270 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:50.268911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:11:50.282 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:11:50.280592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:00.502 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:00.501713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:00.513 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:00.512497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:10.735 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:10.733893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:10.746 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:10.744833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:20.966 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:20.964970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:20.978 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:20.976968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:31.194 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:31.193122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:31.206 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:31.205185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:41.425 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:41.424294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:41.436 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:41.435648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:46.648 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:46.647509 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T16:12:47.337 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:47.336351 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:12:47.324+0000 7f9500242340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:12:47.338 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:12:47.324+0000 7f9500242340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:12:47.338 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:12:47.324+0000 7f9500242340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:12:47.338 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:12:47.339 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:47.336377 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:12:47.324+0000 7f9500242340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:12:47.339 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:12:47.324+0000 7f9500242340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:12:47.339 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:12:47.324+0000 7f9500242340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:12:47.339 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:12:51.657 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:51.656128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:12:51.667 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:12:51.666436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:01.884 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:01.883533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:01.897 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:01.896195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:12.123 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:12.122031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:12.201 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:12.200684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:22.484 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:22.483121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:22.493 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:22.492657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:22.752 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:22.751833 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 16:13:22.742770648 +0000 UTC m=+4751.464059488 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T16:13:32.713 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:32.712158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:32.726 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:32.725538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:42.943 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:42.942749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:42.951 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:42.950634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:53.187 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:53.186733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:13:53.200 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:13:53.199266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:03.417 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:03.416285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:03.430 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:03.429515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:13.651 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:13.650307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:13.663 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:13.662251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:23.880 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:23.879563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:23.932 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:23.931100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:34.155 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:34.154692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:34.168 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:34.167678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:44.387 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:44.386432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:44.400 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:44.400210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:54.628 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:54.626525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:14:54.638 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:14:54.637786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:04.859 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:04.858473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:04.870 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:04.869343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:15.112 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:15.111443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:15.126 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:15.125771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:25.346 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:25.344550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:25.359 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:25.357543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:35.622 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:35.621607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:35.635 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:35.634648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:45.854 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:45.853045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:45.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:45.863917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:56.101 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:56.100597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:15:56.112 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:15:56.111763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:06.332 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:06.330934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:06.346 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:06.345754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:16.567 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:16.566185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:16.578 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:16.578194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:26.815 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:26.814720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:26.826 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:26.825876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:37.105 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:37.102765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:37.116 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:37.114993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:47.351 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:47.350768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:47.374 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:47.360320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:57.578 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:57.576828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:16:57.590 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:16:57.589480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:07.812 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:07.811343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:07.824 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:07.822805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:18.043 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:18.042105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:18.128 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:18.052784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:28.272 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:28.271530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:28.280 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:28.280248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:38.503 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:38.502563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:38.516 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:38.515284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:48.739 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:48.738018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:48.754 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:48.753866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:58.972 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:58.970974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:17:58.985 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:17:58.984492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:09.221 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:09.220744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:09.234 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:09.233099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:19.457 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:19.456217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:19.468 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:19.467902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:29.689 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:29.687745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:29.703 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:29.702383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:39.921 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:39.920413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:39.934 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:39.934122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:50.156 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:50.154553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:18:50.167 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:18:50.165463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:00.405 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:00.404325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:00.415 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:00.414243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:10.637 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:10.636043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:10.648 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:10.646851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:20.876 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:20.875611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:20.884 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:20.883806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:31.126 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:31.125444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:31.138 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:31.137139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:41.358 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:41.357317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:41.371 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:41.370834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:51.589 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:51.588616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:19:51.604 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:19:51.602706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:01.845 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:01.844708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:01.854 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:01.852713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:12.072 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:12.070800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:12.082 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:12.081168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:22.304 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:22.303640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:22.315 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:22.314997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:32.541 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:32.540347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:32.565 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:32.564384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:42.812 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:42.810976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:42.829 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:42.828974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:53.052 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:53.050865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:20:53.060 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:20:53.059477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:03.302 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:03.301386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:03.314 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:03.313452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:13.542 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:13.541656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:13.555 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:13.554199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:23.791 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:23.789833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:23.824 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:23.823046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:34.047 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:34.046163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:34.059 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:34.058432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:44.277 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:44.275802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:44.290 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:44.289225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:54.526 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:54.524788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:21:54.539 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:21:54.538209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:04.767 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:04.765911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:04.782 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:04.782011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:15.006 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:15.005849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:15.014 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:15.013615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:25.234 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:25.233567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:25.247 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:25.246679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:35.466 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:35.465231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:35.478 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:35.477385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:45.711 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:45.709923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:45.723 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:45.722571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:55.959 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:55.958575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:22:55.968 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:22:55.967332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:06.189 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:06.187503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:06.235 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:06.234068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:16.497 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:16.496748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:16.534 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:16.533162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:26.788 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:26.786758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:26.799 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:26.798863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:37.063 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:37.062225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:37.074 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:37.073847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:47.309 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:47.308532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:47.321 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:47.320582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:57.538 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:57.537306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:23:57.550 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:23:57.549766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:07.763 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:07.762498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:07.774 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:07.773478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:17.994 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:17.992803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:18.002 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:18.001744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:28.287 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:28.285851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:28.298 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:28.297721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:38.521 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:38.520446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:38.534 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:38.533433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:48.752 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:48.751556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:48.762 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:48.761744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:59.023 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:59.022223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:24:59.035 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:24:59.034443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:09.252 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:09.251077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:09.265 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:09.264136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:19.484 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:19.483259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:19.496 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:19.495303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:29.747 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:29.745638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:29.759 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:29.758712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:39.979 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:39.978250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:39.991 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:39.990443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:50.212 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:50.211544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:25:50.225 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:25:50.224212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:00.472 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:00.470892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:00.483 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:00.481654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:10.704 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:10.703440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:10.718 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:10.715814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:20.936 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:20.935552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:20.948 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:20.947125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:31.164 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:31.162886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:31.177 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:31.176107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:41.400 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:41.399431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:41.414 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:41.413812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:51.639 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:51.638136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:26:51.651 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:26:51.650296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:01.870 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:01.869559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:01.882 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:01.881320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:12.104 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:12.103584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:12.118 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:12.117286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:22.402 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:22.400959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:22.420 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:22.419684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:32.647 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:32.646743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:32.662 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:32.660837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:32.662 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:32.660881 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:42.876 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:42.875491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:42.895 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:42.894166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:53.115 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:53.113879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:27:53.128 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:27:53.128086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:03.350 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:03.349054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:03.361 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:03.360392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:13.584 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:13.582667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:13.595 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:13.594055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:23.814 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:23.813434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:23.868 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:23.868007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:34.168 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:34.167121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:34.206 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:34.204715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:44.449 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:44.448504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:44.461 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:44.460172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:54.679 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:54.678383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:28:54.694 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:28:54.691767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:04.911 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:04.910561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:04.922 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:04.921676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:15.182 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:15.180874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:15.191 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:15.190485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:25.412 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:25.411412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:25.425 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:25.424041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:27.350 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:27.348986 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T16:29:27.942 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:27.940926 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:29:27.933+0000 7f1148ff9340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:29:27.943 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:29:27.933+0000 7f1148ff9340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:29:27.943 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:29:27.933+0000 7f1148ff9340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:29:27.943 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:29:27.943 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:27.940949 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:29:27.933+0000 7f1148ff9340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:29:27.944 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:29:27.933+0000 7f1148ff9340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:29:27.944 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:29:27.933+0000 7f1148ff9340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:29:27.944 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:29:35.646 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:35.644687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:35.698 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:35.697808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:45.935 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:45.934651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:45.950 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:45.949388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:56.167 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:56.165240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:29:56.178 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:29:56.177771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:06.395 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:06.394436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:06.404 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:06.403655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:16.624 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:16.622904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:16.635 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:16.634470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:26.859 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:26.858335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:26.873 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:26.872357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:37.095 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:37.094388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:37.107 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:37.106068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:47.332 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:47.330503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:47.343 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:47.342590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:57.560 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:57.559695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:30:57.573 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:30:57.572087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:07.801 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:07.800190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:07.816 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:07.815282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:18.041 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:18.039788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:18.053 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:18.052865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:28.336 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:28.334767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:28.345 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:28.344207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:38.494 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:38.492881 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 16:31:38.4858498 +0000 UTC m=+5847.207138585 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T16:31:38.576 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:38.575123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:38.620 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:38.619683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:48.859 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:48.857780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:48.871 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:48.870535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:59.122 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:59.121664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:31:59.130 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:31:59.130003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:09.353 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:09.351377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:09.423 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:09.400719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:19.641 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:19.640474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:19.655 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:19.654221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:29.893 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:29.891748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:29.901 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:29.900882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:40.121 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:40.120252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:40.135 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:40.133688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:50.355 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:50.354199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:32:50.365 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:32:50.364901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:00.599 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:00.597924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:00.610 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:00.609996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:10.831 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:10.830286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:10.842 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:10.841353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:21.060 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:21.059296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:21.072 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:21.071723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:31.297 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:31.296538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:31.309 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:31.308932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:41.532 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:41.531736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:41.540 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:41.539497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:51.793 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:51.792197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:33:51.805 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:33:51.804143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:02.023 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:02.022226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:02.038 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:02.036930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:12.283 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:12.281616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:12.294 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:12.294006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:22.519 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:22.518616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:22.530 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:22.529657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:32.752 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:32.751290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:32.792 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:32.791697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:43.015 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:43.014569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:43.025 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:43.024510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:53.245 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:53.244282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:34:53.259 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:34:53.258596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:03.481 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:03.480502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:03.494 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:03.493699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:13.716 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:13.715092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:13.725 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:13.723950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:23.948 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:23.947538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:23.959 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:23.958498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:34.176 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:34.174715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:34.188 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:34.187363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:44.428 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:44.427610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:44.440 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:44.439094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:54.710 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:54.709157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:35:54.723 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:35:54.722453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:04.950 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:04.949348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:05.037 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:05.035811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:15.310 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:15.308972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:15.323 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:15.322729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:25.572 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:25.571110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:25.585 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:25.584514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:35.813 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:35.811748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:35.823 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:35.822696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:46.042 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:46.040790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:46.053 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:46.052861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:56.278 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:56.276820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:36:56.291 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:36:56.289802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:06.511 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:06.509876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:06.524 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:06.523647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:16.740 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:16.739115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:16.753 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:16.752143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:26.976 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:26.974058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:26.987 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:26.986361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:37.219 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:37.217789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:37.235 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:37.231398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:47.450 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:47.449455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:47.464 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:47.462614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:57.686 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:57.684886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:37:57.697 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:37:57.696141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:07.916 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:07.915111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:07.928 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:07.926973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:18.152 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:18.150724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:18.257 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:18.256491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:28.546 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:28.544997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:28.560 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:28.559152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:38.779 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:38.777999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:38.793 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:38.791875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:49.013 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:49.011649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:49.026 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:49.025295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:59.249 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:59.248296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:38:59.271 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:38:59.269872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:09.517 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:09.516288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:09.530 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:09.529315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:19.751 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:19.750358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:19.761 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:19.760939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:29.982 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:29.981270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:29.995 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:29.994878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:40.248 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:40.247335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:40.262 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:40.261093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:50.485 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:50.484066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:39:50.497 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:39:50.496380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:00.718 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:00.717002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:00.731 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:00.730028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:11.000 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:10.998765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:11.010 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:11.009035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:21.234 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:21.233158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:21.243 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:21.242910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:31.504 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:31.503130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:31.516 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:31.515040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:41.736 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:41.734717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:41.750 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:41.749301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:51.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:51.972126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:40:51.985 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:40:51.985082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:02.205 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:02.204219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:02.216 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:02.215731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:12.439 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:12.437789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:12.451 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:12.450578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:22.673 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:22.672406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:22.687 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:22.685405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:32.913 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:32.909374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:32.925 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:32.924820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:43.145 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:43.144281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:43.158 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:43.157512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:53.380 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:53.378836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:41:53.391 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:41:53.391104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:03.612 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:03.611713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:03.623 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:03.622326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:13.844 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:13.843302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:13.855 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:13.854283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:24.148 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:24.147581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:24.158 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:24.157435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:34.430 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:34.428785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:34.439 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:34.438546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:44.701 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:44.699933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:44.712 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:44.711300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:54.935 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:54.933707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:42:54.946 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:42:54.945972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:05.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:05.173550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:05.185 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:05.183836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:15.406 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:15.405447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:15.418 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:15.417449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:25.637 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:25.636192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:25.649 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:25.648836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:35.871 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:35.870006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:35.884 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:35.883146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:46.105 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:46.103977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:46.118 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:46.117125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:56.350 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:56.348842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:43:56.363 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:43:56.361366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:06.581 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:06.580232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:06.592 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:06.591107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:16.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:16.863139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:16.876 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:16.875480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:27.094 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:27.093756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:27.104 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:27.104107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:37.325 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:37.324371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:37.335 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:37.333784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:47.557 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:47.556255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:47.568 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:47.567761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:57.824 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:57.823056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:44:57.839 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:44:57.837907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:08.058 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:08.057498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:08.070 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:08.069529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:18.292 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:18.290874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:18.301 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:18.300881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:28.521 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:28.520611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:28.532 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:28.531075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:38.751 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:38.750050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:38.763 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:38.762814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:48.980 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:48.978859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:48.991 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:48.990596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:59.215 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:59.213885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:45:59.226 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:45:59.225345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:07.949 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:07.947696 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T16:46:08.537 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:08.535807 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:46:08.526+0000 7f8597f25340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:46:08.538 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:46:08.526+0000 7f8597f25340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:46:08.538 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:46:08.526+0000 7f8597f25340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:46:08.538 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:46:08.538 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:08.535843 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:46:08.526+0000 7f8597f25340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:46:08.539 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:46:08.526+0000 7f8597f25340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:46:08.539 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T16:46:08.526+0000 7f8597f25340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:46:08.539 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:46:09.446 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:09.445210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:09.455 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:09.454701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:19.677 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:19.676050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:19.690 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:19.689132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:29.915 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:29.913672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:29.927 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:29.926295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:40.146 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:40.145521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:40.159 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:40.158935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:50.376 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:50.374735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:46:50.388 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:46:50.387256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:00.615 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:00.613732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:00.628 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:00.627023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:10.895 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:10.893225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:10.911 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:10.909939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:21.135 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:21.133800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:21.147 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:21.146309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:31.374 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:31.372830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:31.385 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:31.385228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:41.604 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:41.603541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:41.616 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:41.615625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:51.836 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:51.835052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:47:51.846 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:47:51.845711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:02.066 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:02.065266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:02.077 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:02.076867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:12.333 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:12.331767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:12.343 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:12.343133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:22.632 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:22.630872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:22.642 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:22.642045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:32.862 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:32.860838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:32.878 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:32.877804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:43.112 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:43.111252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:43.125 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:43.124834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:53.342 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:53.341322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:48:53.346 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:53.345890 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 16:48:53.33807196 +0000 UTC m=+6882.059360758 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T16:48:53.353 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:48:53.353294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:03.570 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:03.569205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:03.583 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:03.582275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:13.813 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:13.809080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:13.821 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:13.820706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:24.043 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:24.042498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:24.055 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:24.054861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:34.279 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:34.278203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:34.291 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:34.290777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:44.549 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:44.548031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:44.605 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:44.604894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:54.825 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:54.824704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:49:54.837 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:49:54.836641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:05.077 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:05.076002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:05.090 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:05.089325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:15.308 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:15.306636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:15.320 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:15.319574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:25.541 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:25.540323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:25.557 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:25.554568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:35.799 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:35.798586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:35.811 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:35.810395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:46.032 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:46.030619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:46.045 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:46.044536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:56.291 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:56.290465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:50:56.305 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:50:56.304528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:06.524 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:06.523097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:06.535 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:06.535021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:16.766 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:16.765259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:16.776 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:16.776100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:27.004 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:27.003646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:27.024 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:27.023125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:37.244 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:37.242724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:37.254 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:37.253882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:47.550 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:47.549177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:47.562 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:47.561858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:57.798 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:57.797041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:51:57.810 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:51:57.809180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:08.045 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:08.044217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:08.057 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:08.056920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:18.277 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:18.276274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:18.289 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:18.288322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:28.533 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:28.532221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:28.545 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:28.544910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:38.766 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:38.765336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:38.799 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:38.796647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:49.042 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:49.040683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:49.053 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:49.052396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:59.275 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:59.274045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:52:59.286 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:52:59.285739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:09.506 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:09.505448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:09.519 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:09.518585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:19.743 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:19.742249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:19.756 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:19.755831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:29.976 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:29.975956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:29.989 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:29.986750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:40.224 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:40.222748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:40.241 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:40.240199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:50.473 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:50.472110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:53:50.482 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:53:50.481764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:00.701 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:00.700558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:00.715 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:00.713995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:10.933 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:10.932656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:10.945 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:10.944485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:21.163 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:21.162285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:21.176 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:21.175165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:31.444 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:31.443088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:31.455 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:31.454106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:41.678 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:41.677153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:41.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:41.690197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:51.915 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:51.913593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:54:51.941 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:54:51.940465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:02.346 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:02.344849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:02.356 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:02.355366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:12.575 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:12.574065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:12.585 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:12.584505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:22.810 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:22.809024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:22.822 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:22.820813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:33.099 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:33.097918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:33.110 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:33.109479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:43.330 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:43.328977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:43.343 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:43.342556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:53.569 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:53.567910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:55:53.580 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:55:53.579556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:03.850 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:03.849676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:03.861 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:03.860675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:14.080 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:14.078650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:14.091 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:14.090604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:24.309 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:24.308465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:24.322 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:24.320875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:34.539 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:34.537927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:34.549 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:34.549059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:44.765 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:44.764373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:44.779 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:44.776550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:54.996 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:54.995639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:56:55.010 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:56:55.009401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:05.293 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:05.291938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:05.304 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:05.304057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:15.526 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:15.525274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:15.553 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:15.552382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:25.778 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:25.776743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:25.790 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:25.789647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:36.237 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:36.236749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:36.247 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:36.246638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:46.557 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:46.555454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:46.644 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:46.643229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:56.866 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:56.864943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:57:56.877 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:57:56.876524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:07.164 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:07.163742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:07.177 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:07.176628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:17.526 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:17.525730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:17.539 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:17.538664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:27.803 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:27.801996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:27.818 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:27.816828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:38.168 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:38.167375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:38.205 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:38.203588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:48.736 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:48.734910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:48.748 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:48.747879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:58.969 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:58.968020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:58:58.982 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:58:58.981380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:09.358 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:09.356678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:09.387 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:09.386800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:19.754 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:19.753448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:19.765 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:19.764571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:30.039 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:30.038171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:30.050 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:30.049938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:40.316 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:40.315216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:40.325 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:40.324855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:50.581 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:50.580581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T16:59:50.593 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 16:59:50.593081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:00.809 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:00.808729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:00.821 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:00.820791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:11.066 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:11.065478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:11.078 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:11.077503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:21.394 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:21.393733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:21.403 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:21.402896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:31.621 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:31.619974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:31.632 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:31.631424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:42.015 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:42.014158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:42.028 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:42.027491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:52.299 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:52.297973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:00:52.309 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:00:52.308634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:02.529 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:02.527821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:02.543 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:02.541335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:12.760 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:12.758989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:12.771 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:12.770961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:22.996 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:22.995068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:23.009 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:23.007410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:33.264 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:33.263690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:33.276 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:33.275692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:43.536 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:43.534710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:43.548 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:43.547550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:53.828 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:53.826566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:01:53.841 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:01:53.839594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:04.058 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:04.057403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:04.070 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:04.069188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:14.289 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:14.287552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:14.301 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:14.300879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:24.540 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:24.538927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:24.550 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:24.549894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:34.769 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:34.767955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:34.781 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:34.780227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:45.000 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:44.998816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:45.013 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:45.012634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:48.549 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:48.548214 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T17:02:49.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:49.172227 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:02:49.162+0000 7f9f05628340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:02:49.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:02:49.162+0000 7f9f05628340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:02:49.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:02:49.162+0000 7f9f05628340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:02:49.175 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:02:49.175 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:49.172267 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:02:49.162+0000 7f9f05628340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:02:49.175 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:02:49.162+0000 7f9f05628340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:02:49.176 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:02:49.162+0000 7f9f05628340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:02:49.176 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:02:55.233 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:55.232526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:02:55.244 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:02:55.242882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:05.459 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:05.458845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:05.468 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:05.467564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:05.526 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:05.525280 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 17:03:05.515707479 +0000 UTC m=+7734.236996267 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T17:03:15.698 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:15.697056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:15.709 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:15.708956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:25.924 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:25.923282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:25.937 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:25.936641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:36.155 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:36.154620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:36.166 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:36.165121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:46.404 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:46.402724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:46.431 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:46.430395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:56.684 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:56.682953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:03:56.695 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:03:56.694313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:06.955 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:06.914290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:06.986 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:06.985052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:17.201 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:17.200189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:17.210 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:17.210098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:27.427 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:27.426532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:27.439 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:27.438851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:37.660 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:37.659690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:37.674 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:37.673783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:47.894 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:47.893195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:47.906 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:47.904996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:58.123 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:58.122253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:04:58.137 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:04:58.136639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:08.357 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:08.355994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:08.368 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:08.367153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:18.590 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:18.588771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:18.605 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:18.603388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:28.823 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:28.822462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:28.833 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:28.833265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:39.053 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:39.052693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:39.065 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:39.064830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:49.307 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:49.306380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:49.330 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:49.329833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:59.589 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:59.587835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:05:59.599 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:05:59.598467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:09.827 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:09.826567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:09.839 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:09.838808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:20.058 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:20.057311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:20.072 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:20.071340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:30.292 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:30.291798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:30.305 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:30.303955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:40.556 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:40.554725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:40.569 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:40.567575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:50.789 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:50.787810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:06:50.815 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:06:50.814026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:01.033 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:01.032644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:01.047 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:01.046463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:11.271 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:11.269864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:11.333 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:11.332401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:21.556 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:21.555245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:21.574 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:21.573581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:31.792 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:31.791325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:31.805 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:31.804795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:42.023 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:42.021963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:42.037 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:42.036453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:52.258 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:52.257538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:07:52.272 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:07:52.271094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:02.493 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:02.491694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:02.508 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:02.507705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:12.728 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:12.726769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:12.740 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:12.739707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:22.988 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:22.987462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:23.003 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:23.000608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:33.222 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:33.220805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:33.235 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:33.233525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:43.454 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:43.453507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:43.468 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:43.467854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:53.688 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:53.687113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:08:53.702 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:08:53.701122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:03.923 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:03.922022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:03.936 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:03.935522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:14.160 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:14.159319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:14.173 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:14.172443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:24.400 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:24.398886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:24.410 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:24.409200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:34.635 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:34.633848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:34.647 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:34.646267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:44.903 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:44.901918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:44.914 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:44.912845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:55.135 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:55.133696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:09:55.147 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:09:55.146315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:05.367 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:05.366162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:05.388 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:05.387798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:15.608 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:15.606761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:15.675 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:15.674128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:25.928 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:25.922566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:25.935 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:25.934915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:36.167 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:36.166455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:36.179 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:36.178834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:46.400 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:46.398728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:46.410 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:46.409984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:56.670 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:56.669449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:10:56.683 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:10:56.682273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:06.902 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:06.900847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:06.914 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:06.914020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:17.137 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:17.136025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:17.149 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:17.148216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:27.370 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:27.369420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:27.383 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:27.382356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:37.602 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:37.601553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:37.613 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:37.613320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:47.837 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:47.836302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:47.848 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:47.847714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:58.071 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:58.069856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:11:58.085 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:11:58.082968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:08.306 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:08.305207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:08.318 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:08.317923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:18.542 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:18.541459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:18.554 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:18.553919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:28.785 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:28.784036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:28.802 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:28.801549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:39.019 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:39.018276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:39.026 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:39.025513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:49.240 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:49.239492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:49.253 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:49.251998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:59.474 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:59.473370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:12:59.483 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:12:59.483002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:09.704 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:09.703390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:09.716 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:09.715937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:19.948 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:19.942921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:19.959 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:19.958467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:30.182 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:30.181375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:30.195 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:30.193949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:40.440 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:40.439454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:40.460 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:40.459624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:50.681 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:50.680111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:13:50.692 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:13:50.691484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:00.920 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:00.919398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:00.933 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:00.931886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:11.153 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:11.151907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:11.164 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:11.163469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:21.386 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:21.385680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:21.398 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:21.397498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:31.621 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:31.620388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:31.632 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:31.631795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:41.853 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:41.852149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:41.866 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:41.865047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:52.103 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:52.102665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:14:52.117 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:14:52.116184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:02.339 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:02.338090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:02.351 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:02.350744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:12.573 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:12.572008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:12.582 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:12.581498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:22.801 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:22.800360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:22.816 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:22.815116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:33.055 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:33.054703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:33.068 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:33.067760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:43.292 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:43.290825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:43.303 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:43.302655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:53.525 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:53.523910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:15:53.536 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:15:53.535594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:03.758 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:03.756778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:03.771 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:03.770046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:13.991 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:13.990643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:14.000 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:13.999242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:24.218 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:24.217605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:24.232 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:24.231181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:34.455 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:34.453695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:34.467 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:34.466474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:44.687 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:44.686297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:44.699 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:44.698131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:54.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:54.971976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:16:54.981 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:16:54.980225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:05.201 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:05.200867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:05.213 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:05.212432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:15.433 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:15.427909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:15.441 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:15.441131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:25.664 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:25.662919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:25.677 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:25.676235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:35.898 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:35.897668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:35.913 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:35.912346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:46.151 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:46.150028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:46.161 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:46.160550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:56.419 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:56.418542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:17:56.448 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:17:56.447165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:06.670 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:06.669265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:06.684 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:06.681978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:16.918 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:16.916792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:16.931 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:16.930539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:27.178 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:27.177273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:27.193 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:27.192511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:37.434 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:37.433471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:37.444 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:37.443545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:47.661 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:47.660309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:47.669 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:47.668759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:57.890 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:57.889558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:18:57.904 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:18:57.902366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:08.172 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:08.171600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:08.184 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:08.183420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:18.404 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:18.403540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:18.420 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:18.418663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:28.650 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:28.649002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:28.685 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:28.684352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:29.195 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:29.194597 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T17:19:29.887 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:29.884756 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:19:29.875+0000 7f6b755ee340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:19:29.888 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:19:29.875+0000 7f6b755ee340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:19:29.888 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:19:29.875+0000 7f6b755ee340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:19:29.889 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:19:29.889 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:29.884785 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:19:29.875+0000 7f6b755ee340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:19:29.890 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:19:29.875+0000 7f6b755ee340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:19:29.890 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:19:29.875+0000 7f6b755ee340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:19:29.891 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:19:38.904 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:38.902940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:38.983 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:38.983151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:49.273 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:49.272376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:49.306 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:49.305989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:59.526 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:59.525174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:19:59.538 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:19:59.537980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:09.760 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:09.758718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:09.769 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:09.768391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:20.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:20.173011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:20.193 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:20.192335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:20.584 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:20.583302 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 17:20:20.571982629 +0000 UTC m=+8769.293271428 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T17:20:30.413 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:30.412416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:30.425 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:30.424636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:40.656 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:40.655675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:40.666 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:40.665024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:50.888 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:50.887273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:20:50.899 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:20:50.898625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:01.178 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:01.177733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:01.193 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:01.192852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:11.414 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:11.413278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:11.426 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:11.425548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:21.467 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:21.466500 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 17:21:21.438400098 +0000 UTC m=+8830.159688873 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T17:21:21.649 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:21.648886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:21.662 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:21.661770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:31.882 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:31.881441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:31.898 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:31.897500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:42.130 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:42.129031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:42.143 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:42.143118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:52.370 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:52.368973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:21:52.382 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:21:52.381686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:02.601 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:02.600816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:02.609 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:02.609364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:12.867 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:12.866183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:12.884 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:12.878886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:23.094 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:23.093362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:23.111 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:23.110359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:33.335 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:33.334198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:33.351 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:33.350375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:43.615 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:43.614171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:43.959 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:43.626592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:53.911 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:53.909731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:22:53.919 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:22:53.918920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:04.135 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:04.133820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:04.147 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:04.146243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:14.366 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:14.365207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:14.383 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:14.379712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:24.598 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:24.597334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:24.616 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:24.615649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:34.840 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:34.839178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:34.882 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:34.869611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:45.089 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:45.088384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:45.098 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:45.098047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:55.329 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:55.328440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:23:55.342 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:23:55.341835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:05.563 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:05.561990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:05.576 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:05.575767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:15.798 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:15.796740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:15.828 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:15.809746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:26.031 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:26.030111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:26.043 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:26.042572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:36.263 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:36.262494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:36.274 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:36.274027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:46.492 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:46.491017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:46.545 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:46.503492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:56.723 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:56.722037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:24:56.735 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:24:56.734128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:06.952 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:06.951465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:06.961 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:06.960157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:17.180 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:17.179106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:17.331 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:17.193698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:27.447 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:27.445927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:27.455 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:27.455092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:37.675 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:37.673385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:37.687 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:37.685922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:47.909 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:47.908505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:47.921 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:47.921177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:58.181 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:58.180076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:25:58.198 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:25:58.197553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:08.418 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:08.416774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:08.429 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:08.428431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:18.669 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:18.668446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:18.686 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:18.686053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:28.905 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:28.904746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:28.915 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:28.914341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:39.150 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:39.148429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:39.169 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:39.168629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:49.430 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:49.428384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:49.442 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:49.441585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:59.665 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:59.664380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:26:59.702 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:26:59.700735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:09.937 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:09.936553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:09.951 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:09.950519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:20.172 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:20.169842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:20.183 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:20.182522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:30.401 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:30.400443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:30.414 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:30.413583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:40.628 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:40.627171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:40.640 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:40.639226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:40.640 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:40.639259 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:50.870 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:50.865933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:27:50.892 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:27:50.890896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:01.194 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:01.193175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:01.211 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:01.210284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:11.428 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:11.427050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:11.439 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:11.438342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:21.700 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:21.698903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:21.713 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:21.712173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:32.069 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:32.068047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:32.080 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:32.078957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:42.313 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:42.312333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:42.323 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:42.322185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:52.544 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:52.543543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:28:52.553 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:28:52.552696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:03.115 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:03.114414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:03.148 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:03.145238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:13.365 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:13.363721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:13.377 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:13.376178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:23.596 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:23.595705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:23.609 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:23.608358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:34.147 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:34.146581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:34.159 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:34.157655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:44.379 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:44.377990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:44.391 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:44.390669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:54.669 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:54.667819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:29:54.743 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:29:54.740770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:05.101 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:05.100475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:05.112 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:05.111733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:15.400 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:15.399373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:15.413 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:15.412465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:25.631 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:25.630631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:25.645 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:25.644858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:36.113 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:36.112507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:36.125 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:36.124857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:46.348 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:46.347236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:46.361 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:46.361079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:56.625 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:56.624279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:30:56.634 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:30:56.633173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:07.232 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:07.231366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:07.243 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:07.241873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:17.466 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:17.465415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:17.475 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:17.474271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:27.721 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:27.720303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:27.733 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:27.732522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:38.282 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:38.281780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:38.292 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:38.292264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:48.515 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:48.513968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:48.525 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:48.524939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:58.748 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:58.747477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:31:58.762 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:31:58.761325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:09.261 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:09.259950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:09.269 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:09.268767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:19.490 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:19.488561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:19.502 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:19.501531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:29.727 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:29.726230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:29.781 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:29.780236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:40.076 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:40.073714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:40.084 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:40.083961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:50.346 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:50.345482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:32:50.358 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:32:50.358347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:00.676 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:00.671350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:00.696 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:00.696149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:10.918 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:10.916441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:10.928 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:10.927170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:21.149 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:21.148481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:21.163 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:21.162262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:31.387 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:31.385827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:31.400 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:31.399785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:41.622 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:41.621158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:41.634 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:41.633020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:51.858 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:51.856835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:33:51.870 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:33:51.869607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:02.089 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:02.088299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:02.102 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:02.101623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:12.324 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:12.322868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:12.334 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:12.333678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:22.554 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:22.553592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:22.565 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:22.564959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:32.697 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:32.696234 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 17:34:32.687441511 +0000 UTC m=+9621.408730301 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T17:34:32.779 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:32.777955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:32.789 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:32.787110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:43.051 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:43.049901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:43.062 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:43.061506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:53.281 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:53.280397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:34:53.293 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:34:53.293097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:03.515 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:03.513957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:03.527 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:03.526827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:13.750 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:13.749460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:13.763 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:13.762507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:23.981 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:23.980219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:23.993 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:23.992047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:34.212 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:34.211009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:34.224 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:34.224037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:44.486 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:44.485335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:44.500 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:44.499548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:54.719 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:54.718092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:35:54.784 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:35:54.779699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:05.030 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:05.029719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:05.043 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:05.042841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:09.891 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:09.890601 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T17:36:10.507 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:10.505787 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:36:10.496+0000 7f2c22d45340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:36:10.507 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:36:10.496+0000 7f2c22d45340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:36:10.508 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:36:10.496+0000 7f2c22d45340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:36:10.508 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:36:10.508 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:10.505812 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:36:10.496+0000 7f2c22d45340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:36:10.508 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:36:10.496+0000 7f2c22d45340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:36:10.509 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:36:10.496+0000 7f2c22d45340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:36:10.509 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:36:15.267 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:15.265944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:15.279 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:15.279053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:25.531 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:25.530317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:25.543 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:25.542751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:35.788 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:35.786990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:35.800 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:35.799476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:46.018 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:46.017116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:46.029 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:46.028914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:56.247 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:56.245757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:36:56.258 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:36:56.257316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:06.524 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:06.523546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:06.537 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:06.536768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:16.781 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:16.780104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:16.789 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:16.788929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:27.009 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:27.008481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:27.021 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:27.020931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:37.300 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:37.298810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:37.313 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:37.312853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:47.531 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:47.529633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:47.542 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:47.541966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:57.758 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:57.757020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:37:57.773 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:37:57.772194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:07.996 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:07.995056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:08.008 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:08.007433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:18.235 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:18.234068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:18.248 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:18.248012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:28.472 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:28.471057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:28.484 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:28.483956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:38.711 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:38.710573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:38.724 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:38.723641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:48.957 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:48.956811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:48.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:48.972322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:59.221 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:59.220336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:38:59.233 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:38:59.232512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:09.451 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:09.449769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:09.467 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:09.466358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:19.690 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:19.688849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:19.703 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:19.702030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:29.924 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:29.922767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:29.934 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:29.934094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:40.156 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:40.155615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:40.167 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:40.166676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:50.383 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:50.381778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:39:50.434 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:39:50.433510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:00.655 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:00.654416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:00.665 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:00.664144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:10.881 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:10.880429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:10.897 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:10.894924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:21.118 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:21.117086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:21.164 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:21.163318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:31.418 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:31.417114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:31.431 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:31.429822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:41.656 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:41.655499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:41.666 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:41.665925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:51.881 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:51.880076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:40:51.893 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:40:51.892095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:02.144 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:02.143238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:02.153 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:02.152778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:12.411 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:12.409704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:12.421 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:12.420617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:22.703 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:22.701630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:22.714 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:22.713190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:32.939 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:32.935637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:32.951 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:32.946368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:43.213 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:43.212367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:43.228 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:43.227656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:53.453 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:53.452459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:41:53.465 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:41:53.464498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:03.682 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:03.681362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:03.695 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:03.694696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:13.915 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:13.914573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:13.928 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:13.927064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:24.146 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:24.145389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:24.159 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:24.158572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:34.380 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:34.379337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:34.392 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:34.391836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:44.615 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:44.613497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:44.626 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:44.625910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:54.849 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:54.848034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:42:54.859 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:42:54.858794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:05.090 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:05.089358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:05.106 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:05.105392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:15.332 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:15.330837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:15.342 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:15.342083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:25.568 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:25.566896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:25.578 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:25.577549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:35.801 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:35.800307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:35.853 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:35.851884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:46.076 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:46.075423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:46.091 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:46.089825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:56.305 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:56.304302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:43:56.318 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:43:56.316675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:06.535 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:06.534087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:06.549 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:06.548437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:16.798 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:16.796926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:16.810 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:16.809468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:27.026 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:27.024908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:27.037 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:27.036746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:37.281 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:37.279688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:37.292 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:37.291337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:47.509 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:47.507712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:47.589 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:47.587885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:57.816 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:57.814866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:44:57.827 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:44:57.826221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:08.056 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:08.055044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:08.069 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:08.068874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:18.291 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:18.290159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:18.302 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:18.301474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:28.522 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:28.521500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:28.535 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:28.534621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:38.759 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:38.758211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:38.768 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:38.767855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:49.022 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:49.021744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:49.035 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:49.034515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:59.246 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:59.245843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:45:59.257 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:45:59.256594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:09.471 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:09.470591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:09.485 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:09.483858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:19.704 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:19.702758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:19.718 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:19.717156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:29.940 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:29.938777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:29.953 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:29.952476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:40.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:40.173369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:40.216 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:40.215169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:50.437 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:50.436304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:46:50.452 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:46:50.451285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:00.712 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:00.711518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:00.728 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:00.723783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:10.978 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:10.977100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:10.989 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:10.987932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:21.334 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:21.333136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:21.360 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:21.344584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:31.566 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:31.565059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:31.579 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:31.577968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:41.798 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:41.797357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:41.811 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:41.810152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:52.025 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:52.024326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:47:52.036 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:47:52.035460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:02.287 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:02.285603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:02.298 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:02.297994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:12.526 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:12.524574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:12.544 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:12.543907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:22.766 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:22.764585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:22.778 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:22.777489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:32.996 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:32.995180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:33.007 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:33.006805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:43.263 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:43.261437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:43.275 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:43.273108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:53.494 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:53.493329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:48:53.507 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:48:53.506559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:03.726 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:03.725707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:03.736 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:03.735217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:13.986 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:13.985106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:13.997 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:13.996803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:24.217 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:24.216460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:24.228 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:24.226245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:34.441 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:34.440313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:34.453 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:34.452071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:44.671 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:44.669830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:44.683 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:44.682175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:54.904 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:54.903090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:49:54.916 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:49:54.915937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:05.167 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:05.166145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:05.180 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:05.179359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:15.431 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:15.430746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:15.444 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:15.443416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:25.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:25.690118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:25.702 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:25.701534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:35.924 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:35.922920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:35.937 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:35.935383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:46.152 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:46.150958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:46.162 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:46.161490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:56.381 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:56.380147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:50:56.393 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:50:56.392777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:06.616 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:06.615699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:06.628 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:06.627164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:16.846 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:16.845043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:16.856 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:16.855935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:27.088 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:27.087282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:27.108 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:27.107515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:37.369 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:37.368059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:37.381 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:37.380778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:47.548 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:47.546920 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 17:51:47.536966785 +0000 UTC m=+10656.258255564 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T17:51:47.600 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:47.599166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:47.613 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:47.612579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:57.834 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:57.832643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:51:57.846 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:51:57.845404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:08.067 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:08.065422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:08.080 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:08.079632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:18.295 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:18.294424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:18.309 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:18.307920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:28.533 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:28.531647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:28.546 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:28.545596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:38.762 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:38.759828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:38.840 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:38.839412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:49.107 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:49.105988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:49.121 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:49.120250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:50.517 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:50.516028 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T17:52:51.156 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:51.154430 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:52:51.145+0000 7f0dbfdad340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:52:51.156 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:52:51.145+0000 7f0dbfdad340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:52:51.156 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:52:51.145+0000 7f0dbfdad340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:52:51.157 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:52:51.157 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:51.154456 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:52:51.145+0000 7f0dbfdad340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:52:51.157 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:52:51.145+0000 7f0dbfdad340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:52:51.157 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T17:52:51.145+0000 7f0dbfdad340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:52:51.158 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:52:59.342 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:59.341542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:52:59.355 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:52:59.354824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:09.576 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:09.575551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:09.588 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:09.587809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:19.808 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:19.807497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:19.822 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:19.820894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:30.041 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:30.040240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:30.053 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:30.052066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:40.277 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:40.275832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:40.288 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:40.287541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:50.511 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:50.510140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:53:50.520 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:53:50.520032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:00.740 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:00.739618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:00.754 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:00.752467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:10.969 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:10.967925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:10.978 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:10.977301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:21.234 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:21.233222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:21.243 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:21.242573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:31.463 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:31.461984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:31.476 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:31.475754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:41.715 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:41.713605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:41.728 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:41.727369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:51.953 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:51.952101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:54:51.964 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:54:51.963348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:02.183 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:02.182569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:02.194 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:02.193837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:12.415 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:12.410437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:12.423 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:12.422812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:22.657 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:22.656657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:22.669 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:22.668174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:32.888 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:32.887265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:32.935 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:32.934248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:43.155 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:43.153740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:43.164 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:43.164093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:53.398 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:53.397128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:55:53.414 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:55:53.412656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:03.654 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:03.652651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:03.666 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:03.664519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:13.886 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:13.884973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:13.895 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:13.894855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:24.131 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:24.130045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:24.145 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:24.144381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:34.396 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:34.395634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:34.411 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:34.409997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:44.630 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:44.628929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:44.643 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:44.642407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:54.865 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:54.863994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:56:54.874 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:56:54.873861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:05.094 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:05.093137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:05.104 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:05.103133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:15.324 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:15.323299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:15.336 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:15.335280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:25.721 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:25.720559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:25.737 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:25.736687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:35.954 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:35.953586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:35.964 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:35.963467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:46.188 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:46.186570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:46.201 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:46.198924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:56.422 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:56.421347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:57:56.436 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:57:56.435035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:06.660 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:06.658721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:06.672 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:06.671606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:16.894 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:16.893407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:16.905 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:16.904742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:27.196 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:27.194750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:27.205 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:27.204698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:37.468 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:37.467159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:37.486 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:37.484903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:47.754 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:47.753133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:47.764 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:47.763289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:58.232 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:58.230695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:58:58.267 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:58:58.266820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:08.481 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:08.479799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:08.491 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:08.490771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:18.850 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:18.848692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:18.863 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:18.862705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:29.194 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:29.193491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:29.201 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:29.201104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:39.418 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:39.416849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:39.427 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:39.426583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:49.682 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:49.680938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:49.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:49.690618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:59.910 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:59.909392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T17:59:59.918 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 17:59:59.916960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:10.157 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:10.156460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:10.167 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:10.166642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:20.482 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:20.480931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:20.491 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:20.489996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:30.738 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:30.737542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:30.753 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:30.749723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:40.973 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:40.972028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:40.986 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:40.985244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:51.211 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:51.209705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:00:51.226 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:00:51.225412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:01.452 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:01.450880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:01.464 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:01.463667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:11.686 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:11.685189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:11.698 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:11.697041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:21.933 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:21.932333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:21.950 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:21.949068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:32.175 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:32.173941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:32.188 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:32.187126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:42.421 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:42.419741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:42.433 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:42.432017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:52.656 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:52.655150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:01:52.667 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:01:52.666090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:02.912 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:02.911480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:02.924 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:02.923819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:13.159 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:13.157527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:13.170 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:13.169231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:23.399 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:23.397787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:23.407 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:23.406834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:33.649 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:33.647695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:33.664 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:33.662307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:43.887 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:43.884277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:43.900 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:43.899556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:54.144 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:54.142941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:02:54.157 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:02:54.156030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:04.429 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:04.428270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:04.441 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:04.440977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:14.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:14.690464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:14.705 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:14.702296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:24.917 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:24.916517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:24.930 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:24.929208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:35.149 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:35.147500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:35.169 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:35.168047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:45.388 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:45.387467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:45.396 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:45.395379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:55.628 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:55.626757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:03:55.639 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:03:55.638537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:05.993 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:05.992651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:06.005 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:06.004653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:16.340 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:16.339125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:16.354 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:16.353903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:26.587 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:26.586493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:26.601 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:26.600583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:36.824 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:36.823515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:36.836 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:36.834930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:47.056 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:47.055009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:47.073 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:47.072863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:57.295 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:57.294120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:04:57.306 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:04:57.305869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:07.521 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:07.520470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:07.530 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:07.529799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:17.747 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:17.745674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:17.757 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:17.756898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:27.981 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:27.979530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:27.993 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:27.991771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:38.229 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:38.228748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:38.239 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:38.238402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:48.461 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:48.460225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:48.474 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:48.473567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:58.697 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:58.696695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:05:58.709 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:05:58.708578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:09.102 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:09.100791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:09.113 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:09.112220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:19.353 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:19.352039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:19.365 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:19.364409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:29.581 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:29.580293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:29.595 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:29.594566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:39.847 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:39.846596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:39.860 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:39.859500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:50.089 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:50.088038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:06:50.125 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:06:50.123933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:00.340 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:00.338843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:00.381 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:00.379919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:00.728 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:00.727608 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 18:07:00.717179378 +0000 UTC m=+11569.438468151 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T18:07:10.601 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:10.600374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:10.614 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:10.612948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:20.835 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:20.833907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:20.849 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:20.847389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:31.075 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:31.073635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:31.085 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:31.084571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:41.302 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:41.300634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:41.313 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:41.312579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:51.531 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:51.530310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:07:51.542 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:07:51.540977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:01.637 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:01.635561 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 18:08:01.596268171 +0000 UTC m=+11630.317556949 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T18:08:01.761 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:01.760528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:01.775 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:01.773913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:11.995 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:11.993970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:12.006 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:12.005052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:22.225 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:22.224465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:22.237 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:22.236069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:32.457 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:32.456206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:32.470 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:32.468781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:42.693 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:42.691895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:42.707 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:42.706063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:52.926 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:52.924668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:08:52.936 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:08:52.935693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:03.158 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:03.156706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:03.180 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:03.179772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:13.407 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:13.405428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:13.418 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:13.417724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:23.638 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:23.637285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:23.649 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:23.648125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:31.162 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:31.160629 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T18:09:31.773 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:31.770593 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:09:31.761+0000 7f58dc939340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:09:31.773 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:09:31.761+0000 7f58dc939340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:09:31.774 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:09:31.761+0000 7f58dc939340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:09:31.775 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:09:31.775 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:31.770617 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:09:31.761+0000 7f58dc939340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:09:31.776 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:09:31.761+0000 7f58dc939340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:09:31.776 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:09:31.761+0000 7f58dc939340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:09:31.777 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:09:33.868 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:33.866791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:33.884 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:33.881763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:44.102 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:44.101416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:44.110 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:44.110047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:54.336 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:54.334200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:09:54.346 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:09:54.343842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:04.562 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:04.561433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:04.574 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:04.572994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:14.845 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:14.843846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:14.860 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:14.858075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:25.080 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:25.078921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:25.092 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:25.091819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:35.314 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:35.312743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:35.324 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:35.323101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:45.547 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:45.546181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:45.560 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:45.559473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:55.786 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:55.785110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:10:55.819 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:10:55.807591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:06.046 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:06.045194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:06.070 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:06.069638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:16.286 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:16.285055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:16.325 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:16.324508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:26.542 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:26.541113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:26.554 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:26.553569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:36.820 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:36.818731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:36.836 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:36.829379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:47.107 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:47.105605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:47.116 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:47.115171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:57.356 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:57.354758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:11:57.368 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:11:57.367113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:07.628 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:07.626900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:07.640 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:07.638833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:18.027 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:18.026379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:18.040 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:18.038901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:28.262 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:28.261224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:28.276 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:28.275730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:38.498 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:38.497240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:38.511 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:38.509833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:48.969 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:48.967890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:48.979 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:48.978622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:59.271 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:59.269254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:12:59.283 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:12:59.282453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:09.503 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:09.501628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:09.577 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:09.576704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:19.980 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:19.978815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:19.993 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:19.992340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:30.225 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:30.223854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:30.237 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:30.235951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:40.455 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:40.454024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:40.466 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:40.465437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:50.694 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:50.692800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:13:50.704 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:13:50.703419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:00.926 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:00.924872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:00.940 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:00.939349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:11.211 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:11.210165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:11.223 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:11.222401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:21.440 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:21.438390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:21.458 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:21.451295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:31.706 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:31.705375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:31.718 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:31.716807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:41.982 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:41.981170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:41.999 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:41.993986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:52.215 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:52.213848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:14:52.226 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:14:52.225285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:02.448 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:02.446716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:02.457 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:02.456515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:12.680 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:12.678881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:12.691 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:12.690125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:22.925 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:22.923492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:22.940 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:22.938249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:33.163 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:33.161888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:33.175 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:33.174448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:43.398 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:43.397417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:43.411 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:43.409799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:53.631 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:53.630300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:15:53.641 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:15:53.640559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:03.860 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:03.858761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:03.874 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:03.872976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:14.102 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:14.101102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:14.114 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:14.113474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:24.340 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:24.338462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:24.350 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:24.349499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:34.572 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:34.570582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:34.584 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:34.583584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:44.837 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:44.836651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:44.847 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:44.846105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:55.067 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:55.065555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:16:55.077 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:16:55.076340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:05.300 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:05.299260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:05.311 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:05.310313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:15.533 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:15.531618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:15.545 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:15.543899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:25.761 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:25.760209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:25.832 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:25.831019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:36.051 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:36.049866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:36.063 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:36.062160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:46.281 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:46.280426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:46.293 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:46.292218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:56.516 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:56.514868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:17:56.530 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:17:56.529095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:06.752 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:06.750483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:06.768 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:06.767420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:17.029 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:17.027751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:17.039 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:17.037905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:27.314 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:27.312666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:27.327 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:27.326021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:37.548 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:37.546830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:37.560 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:37.559500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:47.778 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:47.776921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:47.790 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:47.789602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:58.015 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:58.014234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:18:58.028 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:18:58.027486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:08.244 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:08.242994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:08.254 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:08.253315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:18.530 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:18.529229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:18.539 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:18.537991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:28.793 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:28.791808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:28.804 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:28.803765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:39.020 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:39.019263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:39.032 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:39.030902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:49.253 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:49.251464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:49.265 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:49.264819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:59.487 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:59.485446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:19:59.500 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:19:59.499739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:09.723 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:09.721932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:09.736 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:09.735454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:19.959 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:19.956749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:19.969 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:19.968563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:30.192 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:30.190489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:30.206 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:30.203302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:40.424 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:40.423386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:40.438 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:40.437218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:50.661 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:50.659956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:20:50.676 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:20:50.673831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:00.902 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:00.900645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:00.915 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:00.914415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:11.144 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:11.142840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:11.156 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:11.154683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:21.409 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:21.407994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:21.431 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:21.430571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:31.648 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:31.647099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:31.659 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:31.658762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:41.880 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:41.879115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:41.895 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:41.894253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:52.115 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:52.113287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:21:52.126 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:21:52.125759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:02.351 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:02.350052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:02.364 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:02.363390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:12.583 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:12.581569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:12.594 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:12.593172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:22.815 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:22.813310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:22.825 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:22.823788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:33.041 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:33.039390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:33.055 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:33.054736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:43.298 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:43.297207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:43.310 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:43.309360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:53.546 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:53.545121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:22:53.556 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:22:53.555745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:03.773 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:03.771954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:03.787 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:03.785243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:14.006 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:14.005026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:14.044 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:14.043333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:24.323 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:24.321762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:24.336 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:24.334925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:34.559 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:34.557335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:34.606 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:34.604811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:44.824 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:44.823093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:44.838 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:44.837677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:55.055 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:55.053872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:23:55.067 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:23:55.065971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:05.344 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:05.343072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:05.358 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:05.356606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:15.557 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:15.556389 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 18:24:15.546705118 +0000 UTC m=+12604.267993911 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T18:24:15.648 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:15.646907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:15.659 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:15.657169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:25.880 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:25.878360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:25.891 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:25.889766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:36.119 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:36.118430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:36.130 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:36.129209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:46.359 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:46.358316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:46.369 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:46.368303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:56.640 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:56.638539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:24:56.653 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:24:56.652457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:06.876 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:06.875252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:06.888 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:06.887456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:17.122 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:17.120670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:17.138 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:17.137328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:27.353 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:27.351750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:27.367 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:27.366177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:37.589 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:37.587661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:37.600 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:37.599337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:47.887 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:47.885396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:47.899 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:47.897919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:58.120 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:58.118354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:25:58.132 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:25:58.131153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:08.351 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:08.349881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:08.361 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:08.360616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:11.783 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:11.781690 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T18:26:12.421 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:12.419027 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:26:12.410+0000 7fe85dd42340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:26:12.421 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:26:12.410+0000 7fe85dd42340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:26:12.421 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:26:12.410+0000 7fe85dd42340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:26:12.422 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:26:12.422 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:12.419049 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:26:12.410+0000 7fe85dd42340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:26:12.422 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:26:12.410+0000 7fe85dd42340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:26:12.422 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:26:12.410+0000 7fe85dd42340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:26:12.423 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:26:18.584 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:18.582695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:18.596 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:18.594759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:28.814 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:28.812916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:28.822 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:28.821442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:39.052 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:39.050242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:39.064 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:39.063219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:49.299 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:49.297440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:49.310 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:49.308961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:59.603 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:59.602173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:26:59.616 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:26:59.615115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:09.854 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:09.852524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:09.865 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:09.864259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:20.128 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:20.125927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:20.138 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:20.136936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:30.383 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:30.382114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:30.392 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:30.390981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:40.656 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:40.649494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:40.662 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:40.661040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:40.662 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:40.661063 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:50.884 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:50.882648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:27:50.896 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:27:50.894525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:01.121 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:01.119948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:01.134 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:01.132770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:11.355 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:11.354103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:11.368 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:11.366969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:21.587 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:21.586063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:21.598 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:21.597255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:31.820 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:31.819322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:31.832 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:31.831749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:42.052 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:42.050824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:42.065 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:42.063983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:52.286 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:52.284391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:28:52.298 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:28:52.297830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:02.517 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:02.516114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:02.528 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:02.527447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:12.757 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:12.755349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:12.793 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:12.791962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:23.011 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:23.010278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:23.023 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:23.022014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:33.243 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:33.241511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:33.254 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:33.252826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:43.468 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:43.466709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:43.480 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:43.479379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:53.716 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:53.714636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:29:53.725 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:29:53.724169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:03.948 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:03.946901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:04.002 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:04.001137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:14.219 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:14.217702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:14.229 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:14.228717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:24.464 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:24.463061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:24.475 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:24.474577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:34.695 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:34.693715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:34.706 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:34.704920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:44.931 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:44.930029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:44.938 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:44.937414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:55.158 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:55.157534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:30:55.168 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:30:55.167262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:05.388 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:05.387025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:05.399 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:05.398440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:15.618 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:15.616576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:15.631 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:15.630607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:25.852 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:25.850733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:25.917 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:25.916319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:36.132 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:36.130762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:36.143 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:36.142515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:46.399 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:46.397749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:46.411 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:46.409995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:56.632 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:56.630563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:31:56.681 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:31:56.679939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:06.908 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:06.906941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:06.926 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:06.925579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:17.163 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:17.161597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:17.175 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:17.174307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:27.396 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:27.395229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:27.408 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:27.407605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:37.661 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:37.659530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:37.674 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:37.672971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:47.918 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:47.917004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:47.931 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:47.929705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:58.152 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:58.151267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:32:58.165 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:32:58.163849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:08.405 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:08.403339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:08.413 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:08.412722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:18.642 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:18.641390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:18.656 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:18.654715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:28.876 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:28.873764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:28.911 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:28.909810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:39.134 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:39.132680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:39.145 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:39.144399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:49.367 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:49.365326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:49.421 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:49.420174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:59.639 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:59.637756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:33:59.652 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:33:59.650971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:09.906 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:09.905026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:09.919 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:09.918670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:20.151 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:20.150065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:20.163 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:20.161883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:30.381 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:30.380135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:30.393 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:30.391934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:40.613 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:40.611627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:40.625 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:40.623992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:50.872 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:50.870787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:34:50.883 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:34:50.882468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:01.104 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:01.102656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:01.116 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:01.115681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:11.342 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:11.340312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:11.356 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:11.355238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:21.580 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:21.578813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:21.591 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:21.590081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:31.813 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:31.811638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:31.824 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:31.823261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:42.045 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:42.044105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:42.056 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:42.055146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:52.277 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:52.276469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:35:52.288 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:35:52.287666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:02.508 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:02.506778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:02.520 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:02.519451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:12.739 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:12.738199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:12.750 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:12.749395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:22.972 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:22.971021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:22.983 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:22.982542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:33.206 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:33.205162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:33.229 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:33.228510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:43.451 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:43.450151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:43.494 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:43.492716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:53.715 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:53.713664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:36:53.726 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:36:53.725937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:03.978 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:03.976990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:03.990 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:03.989463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:14.250 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:14.248769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:14.265 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:14.263840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:24.487 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:24.485707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:24.498 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:24.495892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:34.716 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:34.714537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:34.726 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:34.725514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:44.952 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:44.950978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:44.961 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:44.958914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:55.179 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:55.177974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:37:55.189 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:37:55.188174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:05.406 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:05.405323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:05.416 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:05.415693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:15.671 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:15.669660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:15.683 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:15.682253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:25.904 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:25.903078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:25.920 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:25.919334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:36.161 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:36.159786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:36.181 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:36.180061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:46.401 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:46.399463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:46.463 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:46.462006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:56.689 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:56.687317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:38:56.699 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:38:56.698642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:06.919 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:06.918218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:06.933 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:06.931795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:17.162 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:17.160771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:17.173 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:17.172592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:27.391 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:27.390407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:27.400 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:27.399794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:37.629 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:37.627550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:37.640 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:37.639443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:47.892 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:47.890869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:47.905 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:47.904797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:58.126 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:58.124675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:39:58.138 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:39:58.136840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:08.356 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:08.355114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:08.364 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:08.363740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:18.585 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:18.583694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:18.597 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:18.595801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:28.918 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:28.916524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:28.953 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:28.952718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:39.176 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:39.173522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:39.200 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:39.199488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:49.423 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:49.421472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:49.433 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:49.431863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:59.898 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:59.896806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:40:59.909 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:40:59.908536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:10.127 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:10.125678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:10.136 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:10.134894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:20.398 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:20.396642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:20.409 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:20.408217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:30.457 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:30.455349 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 18:41:30.447518587 +0000 UTC m=+13639.168807334 LastTransitionTime:2022-03-20 14:55:56 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T18:41:30.665 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:30.664770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:30.677 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:30.676571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:40.894 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:40.892836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:40.907 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:40.905839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:51.126 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:51.125420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:41:51.136 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:41:51.135779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:01.355 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:01.353755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:01.366 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:01.365165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:11.588 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:11.587144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:11.600 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:11.599050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:21.857 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:21.855314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:21.868 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:21.867551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:32.123 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:32.121616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:32.135 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:32.134303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:42.398 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:42.396929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:42.413 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:42.411951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:52.427 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:52.425603 I | op-mon: parsing mon endpoints: a=172.21.15.134:6789 2022-03-20T18:42:52.641 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:52.640685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:52.653 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:52.651953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:42:53.036 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:53.034678 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:42:53.023+0000 7fe4f765b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:42:53.036 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:42:53.023+0000 7fe4f765b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:42:53.037 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:42:53.023+0000 7fe4f765b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:42:53.037 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:42:53.037 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:42:53.034702 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:42:53.023+0000 7fe4f765b340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:42:53.037 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:42:53.023+0000 7fe4f765b340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:42:53.038 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20T18:42:53.023+0000 7fe4f765b340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:42:53.038 INFO:tasks.rook.operator.smithi134.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:43:02.871 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:02.869724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:02.895 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:02.894575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:13.115 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:13.114282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:13.127 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:13.126008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:23.347 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:23.345914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:23.422 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:23.420926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:33.663 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:33.661944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:33.672 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:33.671279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:43.903 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:43.901985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:43.911 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:43.910339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:54.151 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:54.150295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:43:54.166 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:43:54.164183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:04.390 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:04.389037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:04.470 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:04.468936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:14.694 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:14.693104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:14.704 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:14.703549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:24.938 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:24.936887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:24.948 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:24.947085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:35.228 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:35.226395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:35.241 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:35.239825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:45.457 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:45.455528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:45.507 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:45.506296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:55.802 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:55.801033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:44:55.814 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:44:55.813526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:06.029 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:06.027509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:06.042 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:06.041263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:16.261 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:16.260478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:16.273 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:16.272493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:26.488 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:26.487574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:26.499 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:26.498252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:36.811 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:36.809881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:36.822 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:36.821391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:47.037 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:47.035823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:47.048 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:47.046839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:57.273 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:57.272214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:45:57.283 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:45:57.282572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:07.505 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:07.503847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:07.537 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:07.535820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:17.753 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:17.751714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:17.808 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:17.807296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:28.035 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:28.033898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:28.049 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:28.047976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:38.266 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:38.265165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:38.277 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:38.276444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:48.515 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:48.514134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:48.525 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:48.524344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:58.781 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:58.780151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:46:58.796 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:46:58.795683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:09.034 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:09.032404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:09.044 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:09.042994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:19.269 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:19.268220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:19.290 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:19.288860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:29.517 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:29.515741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:29.528 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:29.527332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:39.780 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:39.778823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:39.792 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:39.790915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:50.011 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:50.010133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:47:50.024 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:47:50.023494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:00.247 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:00.245836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:00.257 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:00.256760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:10.478 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:10.475972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:10.491 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:10.490610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:20.713 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:20.711630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:20.723 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:20.722727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:30.944 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:30.942241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:30.955 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:30.953828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:41.174 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:41.172205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:41.185 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:41.183806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:51.404 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:51.402765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:48:51.414 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:48:51.413586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:01.635 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:01.634201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:01.663 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:01.661500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:11.892 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:11.891128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:11.905 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:11.904293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:22.128 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:22.127512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:22.142 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:22.141118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:32.361 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:32.359548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:32.384 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:32.381725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:42.611 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:42.610244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:42.621 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:42.620425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:52.856 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:52.854714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:49:52.868 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:49:52.866461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:03.083 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:03.081493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:03.098 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:03.094039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:13.323 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:13.321894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:13.336 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:13.335288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:23.586 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:23.585257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:23.597 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:23.596657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:33.816 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:33.815013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:33.830 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:33.827331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:44.103 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:44.101631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:44.116 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:44.114923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:54.346 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:54.344351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:50:54.358 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:50:54.357634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:04.619 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:04.617774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:04.629 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:04.628596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:14.854 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:14.852333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:14.864 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:14.863601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:25.078 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:25.076676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:25.100 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:25.098941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:35.336 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:35.334829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:35.347 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:35.346158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:45.583 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:45.581617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:45.595 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:45.593828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:55.814 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:55.812610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:51:55.827 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:51:55.825265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:06.042 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:06.041025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:06.054 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:06.053291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:16.274 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:16.272355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:16.287 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:16.285865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:26.507 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:26.505844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:26.520 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:26.519422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:36.742 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:36.740775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:36.754 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:36.753221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:46.972 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:46.970542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:46.981 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:46.979153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:57.200 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:57.199257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:52:57.213 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:52:57.212447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:07.435 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:07.434192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:07.448 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:07.447410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:17.729 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:17.727750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:17.743 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:17.740512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:27.966 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:27.964983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:28.026 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:28.025361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:38.267 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:38.265464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:38.279 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:38.277656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:48.507 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:48.505486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:48.521 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:48.520122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:58.737 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:58.736228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:53:58.747 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:53:58.746655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:54:09.017 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:54:09.015560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T18:54:09.026 INFO:tasks.rook.operator.smithi134.stdout:2022-03-20 18:54:09.024905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-03-20T21:15:00.212 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-20T21:15:00.248 DEBUG:teuthology.task.console_log:Killing console logger for smithi134 2022-03-20T21:15:00.249 DEBUG:teuthology.exit:Finished running handlers